[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:17 dist-test-slave-jccs krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:17 dist-test-slave-jccs krb5kdc[399](info): set up 2 sockets
Sep 28 04:48:17 dist-test-slave-jccs krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 04:48:18.508985 392 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:18 dist-test-slave-jccs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:18.613504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39397
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:18.687979 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:18.688158 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:18.688179 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:18.689308 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:18.689345 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:18.689359 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:18.689402 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:18.689429 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:18.690837 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:18.691056 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:18.691278 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:48:18.696388 412 server_base.cc:1034] running on GCE node
W20260928 04:48:18.696426 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:18.696491 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:18.696641 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:18.696857 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:18.697150 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:18.698328 412 hybrid_clock.cc:648] HybridClock initialized: now 1790570898698319 us; error 27 us; skew 500 ppm
Sep 28 04:48:18 dist-test-slave-jccs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:18.700620 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:18.701006 412 webserver.cc:460] Webserver started at http://127.0.98.62:37223/ using document root <none> and password file <none>
I20260928 04:48:18.701295 412 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:18.701355 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:18.701480 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:18.702492 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "580c99e2a5a1445d9b3392fbcb423555"
format_stamp: "Formatted at 2026-09-28 04:48:18 on dist-test-slave-jccs"
server_key: "b3efd0d3508e68e0e4cdb88c79255310"
server_key_iv: "5b9728c93e4e2306bcff2519c357e3b1"
server_key_version: "encryptionkey@0"
I20260928 04:48:18.702858 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "580c99e2a5a1445d9b3392fbcb423555"
format_stamp: "Formatted at 2026-09-28 04:48:18 on dist-test-slave-jccs"
server_key: "b3efd0d3508e68e0e4cdb88c79255310"
server_key_iv: "5b9728c93e4e2306bcff2519c357e3b1"
server_key_version: "encryptionkey@0"
I20260928 04:48:18.704195 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:18.704911 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:18.705080 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:18.705199 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "580c99e2a5a1445d9b3392fbcb423555"
format_stamp: "Formatted at 2026-09-28 04:48:18 on dist-test-slave-jccs"
server_key: "b3efd0d3508e68e0e4cdb88c79255310"
server_key_iv: "5b9728c93e4e2306bcff2519c357e3b1"
server_key_version: "encryptionkey@0"
I20260928 04:48:18.705281 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:18.727185 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:18.729449 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:18.729609 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:18.733284 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39397
I20260928 04:48:18.733343 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39397 every 8 connection(s)
I20260928 04:48:18.733668 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:18.734475 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:18.736560 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555: Bootstrap starting.
I20260928 04:48:18.737102 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:18.737394 480 log.cc:824] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:18.737740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 412
I20260928 04:48:18.737844 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:18.738104 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555: No bootstrap required, opened a new log
I20260928 04:48:18.738510 392 external_mini_cluster.cc:1428] Setting key 99c5faf97aa442cacee792a6530f793a
I20260928 04:48:18.739954 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580c99e2a5a1445d9b3392fbcb423555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39397 } }
I20260928 04:48:18.740103 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:18.740124 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580c99e2a5a1445d9b3392fbcb423555, State: Initialized, Role: FOLLOWER
I20260928 04:48:18.740254 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580c99e2a5a1445d9b3392fbcb423555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39397 } }
I20260928 04:48:18.740316 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:18.740336 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:18.740401 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:18.741245 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580c99e2a5a1445d9b3392fbcb423555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39397 } }
I20260928 04:48:18.741324 480 leader_election.cc:302] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 580c99e2a5a1445d9b3392fbcb423555; no voters:
I20260928 04:48:18.741478 480 leader_election.cc:288] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:18.741496 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:18.741611 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [term 1 LEADER]: Becoming Leader. State: Replica: 580c99e2a5a1445d9b3392fbcb423555, State: Running, Role: LEADER
I20260928 04:48:18.741680 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580c99e2a5a1445d9b3392fbcb423555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39397 } }
I20260928 04:48:18.741726 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:18.741971 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 580c99e2a5a1445d9b3392fbcb423555. Latest consensus state: current_term: 1 leader_uuid: "580c99e2a5a1445d9b3392fbcb423555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580c99e2a5a1445d9b3392fbcb423555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39397 } } }
I20260928 04:48:18.742039 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [sys.catalog]: This master's current role is: LEADER
W20260928 04:48:18.743395 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 04:48:18.743825 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "580c99e2a5a1445d9b3392fbcb423555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "580c99e2a5a1445d9b3392fbcb423555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39397 } } }
W20260928 04:48:18.745078 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:48:18.745090 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:18.745155 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:18.745357 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:18.747156 490 catalog_manager.cc:1347] Generated new cluster ID: 587f5220295949b890e4af7a617e6e26
I20260928 04:48:18.747208 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:48:18 dist-test-slave-jccs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:18.753091 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:18.739298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.0.98.62:39397)
0928 04:48:18.739448 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:18.739456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:18.739716 (+ 260us) server_negotiation.cc:407] Connection header received
0928 04:48:18.741615 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:18.741622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:18.741678 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:18.741798 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:18.743511 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:18.744831 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:18.746044 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:18.746639 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:18.746763 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:18.749938 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:18.749942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:18.749942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:18.749952 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:18.751422 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:18.751829 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:18.751831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:18.751832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:18.751918 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:18.752154 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:18.752155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:18.752155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:18.752384 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:18.752522 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:18.752753 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 04:48:18.752868 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260928 04:48:18.782701 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:18.783358 490 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:18.794737 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 580c99e2a5a1445d9b3392fbcb423555: Generated new TSK 0
I20260928 04:48:18.794937 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:48:18.831395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39397
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:18.926479 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:18.926698 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:18.926735 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:18.927794 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:18.927855 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:18.927919 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:18.929344 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:18.929586 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:18.929792 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:18.934192 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:18.934171 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:18.934341 506 server_base.cc:1034] running on GCE node
W20260928 04:48:18.934183 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:18.934578 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:18.934747 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:18.935915 506 hybrid_clock.cc:648] HybridClock initialized: now 1790570898935907 us; error 25 us; skew 500 ppm
Sep 28 04:48:18 dist-test-slave-jccs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:18.938187 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:48:18.938604 506 webserver.cc:460] Webserver started at http://127.0.98.1:45197/ using document root <none> and password file <none>
I20260928 04:48:18.938823 506 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:18.938886 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:18.939020 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:18.939945 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5a5bfff472640cfa16d02172724d8d0"
format_stamp: "Formatted at 2026-09-28 04:48:18 on dist-test-slave-jccs"
server_key: "bb4ceb175f452ba20692694ec57690a0"
server_key_iv: "692f8b89ed4b9b848574eb4c4975446f"
server_key_version: "encryptionkey@0"
I20260928 04:48:18.940294 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5a5bfff472640cfa16d02172724d8d0"
format_stamp: "Formatted at 2026-09-28 04:48:18 on dist-test-slave-jccs"
server_key: "bb4ceb175f452ba20692694ec57690a0"
server_key_iv: "692f8b89ed4b9b848574eb4c4975446f"
server_key_version: "encryptionkey@0"
I20260928 04:48:18.941572 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 04:48:18.942312 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:18.942467 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:18.942553 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "f5a5bfff472640cfa16d02172724d8d0"
format_stamp: "Formatted at 2026-09-28 04:48:18 on dist-test-slave-jccs"
server_key: "bb4ceb175f452ba20692694ec57690a0"
server_key_iv: "692f8b89ed4b9b848574eb4c4975446f"
server_key_version: "encryptionkey@0"
I20260928 04:48:18.942694 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:18.974340 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:18.976442 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:18.976541 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:18.976691 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:18.976971 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:18.977003 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:18.977041 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:18.977067 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:18.982918 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44451
I20260928 04:48:18.982963 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44451 every 8 connection(s)
I20260928 04:48:18.983230 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:48:18.986152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 506
I20260928 04:48:18.986238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:18.986408 392 external_mini_cluster.cc:1428] Setting key 9166c13d756f01882cb84364ef5cba8a
Sep 28 04:48:18 dist-test-slave-jccs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:18.992686 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:18.983834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36405 (local address 127.0.98.62:39397)
0928 04:48:18.983878 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:18.983880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:18.984046 (+ 166us) server_negotiation.cc:407] Connection header received
0928 04:48:18.984468 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:18.984468 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:18.984476 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:18.984531 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:18.985371 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:18.985881 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:18.987217 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:18.987400 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:18.987451 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:18.989947 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:18.989948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:18.989949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:18.989955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:18.991496 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:18.991825 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:18.991826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:18.991826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:18.991884 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:18.992087 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:18.992089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:18.992089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:18.992233 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:18.992304 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:18.992504 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 04:48:18.992578 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:48:18.992699 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:18.983810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39397 (local address 127.0.98.1:36405)
0928 04:48:18.983945 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:18.983982 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:18.984394 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:18.984577 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:18.984580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:18.985104 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:18.985302 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:18.985306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:18.985952 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:18.985953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:18.987142 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:18.987145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:18.987857 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:18.987858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:18.988024 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:18.988458 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:18.988467 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:18.989864 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:18.991568 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:18.991574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:18.991575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:18.991757 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:18.991946 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:18.991946 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:18.991946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:18.992031 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:18.992298 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:18.992301 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:18.992394 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 04:48:18.992463 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 04:48:18.992563 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260928 04:48:18.993284 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39397
I20260928 04:48:18.993382 635 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:18.993558 635 heartbeater.cc:511] Master 127.0.98.62:39397 requested a full tablet report, sending...
I20260928 04:48:18.994010 444 ts_manager.cc:194] Registered new tserver with Master: f5a5bfff472640cfa16d02172724d8d0 (127.0.98.1:44451)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:48:18.994606 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:48:19.011749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39397
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:19.087539 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:19.087721 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:19.087743 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:19.088857 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:19.088896 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:19.088922 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:19.090368 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:19.090620 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:19.090844 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:19.095438 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:19.095428 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:19.095515 640 server_base.cc:1034] running on GCE node
W20260928 04:48:19.095430 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:19.095772 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:19.095942 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:19.097090 640 hybrid_clock.cc:648] HybridClock initialized: now 1790570899097079 us; error 29 us; skew 500 ppm
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:19.099368 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:48:19.099745 640 webserver.cc:460] Webserver started at http://127.0.98.2:46381/ using document root <none> and password file <none>
I20260928 04:48:19.099942 640 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:19.100001 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:19.100133 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:19.101054 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "57b103f2bc1c47c795dc840bc9401ba0"
format_stamp: "Formatted at 2026-09-28 04:48:19 on dist-test-slave-jccs"
server_key: "06739174e5b4f8397c9d9e557fab1c87"
server_key_iv: "1d55e5b5d757f44f18fd8405ee359f44"
server_key_version: "encryptionkey@0"
I20260928 04:48:19.101388 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "57b103f2bc1c47c795dc840bc9401ba0"
format_stamp: "Formatted at 2026-09-28 04:48:19 on dist-test-slave-jccs"
server_key: "06739174e5b4f8397c9d9e557fab1c87"
server_key_iv: "1d55e5b5d757f44f18fd8405ee359f44"
server_key_version: "encryptionkey@0"
I20260928 04:48:19.102562 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:19.103227 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:19.103381 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:19.103467 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "57b103f2bc1c47c795dc840bc9401ba0"
format_stamp: "Formatted at 2026-09-28 04:48:19 on dist-test-slave-jccs"
server_key: "06739174e5b4f8397c9d9e557fab1c87"
server_key_iv: "1d55e5b5d757f44f18fd8405ee359f44"
server_key_version: "encryptionkey@0"
I20260928 04:48:19.103538 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:19.122360 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:19.123977 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:19.124114 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:19.124302 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:19.124572 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:19.124615 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:19.124652 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:19.124680 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:19.130731 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42771
I20260928 04:48:19.130795 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42771 every 8 connection(s)
I20260928 04:48:19.131132 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:48:19.135556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 640
I20260928 04:48:19.135630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:19.135793 392 external_mini_cluster.cc:1428] Setting key 2c59bb5ecf9ed21356b7b47f558136ad
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:19.140095 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:19.131801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49217 (local address 127.0.98.62:39397)
0928 04:48:19.131865 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:19.131867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:19.132040 (+ 173us) server_negotiation.cc:407] Connection header received
0928 04:48:19.132549 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:19.132550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:19.132561 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:19.132614 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:19.133560 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.134042 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.134982 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.135173 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.135275 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:19.137471 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:19.137473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:19.137473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:19.137481 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:19.138800 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:19.139208 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:19.139209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:19.139209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:19.139267 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:19.139467 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:19.139468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:19.139469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:19.139601 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:19.139665 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:19.139873 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 04:48:19.139970 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:48:19.140122 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:19.131782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39397 (local address 127.0.98.2:49217)
0928 04:48:19.131926 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:19.131966 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:19.132451 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:19.132682 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:19.132686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:19.133288 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:19.133464 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:19.133470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.134131 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.134132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:19.134887 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:19.134888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.135283 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.135284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:19.135397 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:19.135918 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:19.135925 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:19.137378 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:19.138877 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:19.138881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:19.138883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:19.139144 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:19.139329 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:19.139330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:19.139330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:19.139414 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:19.139658 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:19.139661 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:19.139759 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 04:48:19.139830 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:48:19.139953 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260928 04:48:19.140606 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39397
I20260928 04:48:19.140691 769 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:19.140851 769 heartbeater.cc:511] Master 127.0.98.62:39397 requested a full tablet report, sending...
I20260928 04:48:19.141261 444 ts_manager.cc:194] Registered new tserver with Master: 57b103f2bc1c47c795dc840bc9401ba0 (127.0.98.2:42771)
I20260928 04:48:19.141589 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49217
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:48:19.159344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39397
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:19.235457 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:19.235636 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:19.235659 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:19.236733 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:19.236771 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:19.236797 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:19.238178 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:19.238395 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:19.238598 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:19.243116 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:19.243139 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:19.243132 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:19.243424 774 server_base.cc:1034] running on GCE node
I20260928 04:48:19.243579 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:19.243762 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:19.244886 774 hybrid_clock.cc:648] HybridClock initialized: now 1790570899244860 us; error 41 us; skew 500 ppm
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:19.247150 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:48:19.247494 774 webserver.cc:460] Webserver started at http://127.0.98.3:46073/ using document root <none> and password file <none>
I20260928 04:48:19.247684 774 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:19.247721 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:19.247845 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:19.248785 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "9f61e353b00a400d9d3c1e0033397456"
format_stamp: "Formatted at 2026-09-28 04:48:19 on dist-test-slave-jccs"
server_key: "e8b673b310dfc29cd2cb66ff9b92d50e"
server_key_iv: "f32a4ce713969d11841b2f32308c168a"
server_key_version: "encryptionkey@0"
I20260928 04:48:19.249117 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9f61e353b00a400d9d3c1e0033397456"
format_stamp: "Formatted at 2026-09-28 04:48:19 on dist-test-slave-jccs"
server_key: "e8b673b310dfc29cd2cb66ff9b92d50e"
server_key_iv: "f32a4ce713969d11841b2f32308c168a"
server_key_version: "encryptionkey@0"
I20260928 04:48:19.250360 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:19.250980 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:19.251124 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:19.251205 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "9f61e353b00a400d9d3c1e0033397456"
format_stamp: "Formatted at 2026-09-28 04:48:19 on dist-test-slave-jccs"
server_key: "e8b673b310dfc29cd2cb66ff9b92d50e"
server_key_iv: "f32a4ce713969d11841b2f32308c168a"
server_key_version: "encryptionkey@0"
I20260928 04:48:19.251271 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:19.272603 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:19.274302 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:19.274441 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:19.274638 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:19.274910 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:19.274955 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:19.274984 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:19.275008 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:19.281456 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39283
I20260928 04:48:19.281509 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39283 every 8 connection(s)
I20260928 04:48:19.281814 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:48:19.283324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 774
I20260928 04:48:19.283419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:19.283573 392 external_mini_cluster.cc:1428] Setting key c29c59993af5e8b6f8e14cd5b1b8ff24
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:19.290068 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:19.282495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56525 (local address 127.0.98.62:39397)
0928 04:48:19.282534 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:19.282536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:19.282697 (+ 161us) server_negotiation.cc:407] Connection header received
0928 04:48:19.283154 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:19.283155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:19.283179 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:19.283232 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:19.284352 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.284913 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.285878 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.286067 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.286115 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:19.287960 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:19.287961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:19.287962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:19.287967 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:19.289058 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:19.289445 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:19.289445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:19.289446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:19.289491 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:19.289668 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:19.289668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:19.289668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:19.289748 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:19.289788 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:19.289965 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 04:48:19.290013 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:48:19.290239 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:19.282471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39397 (local address 127.0.98.3:56525)
0928 04:48:19.282605 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:19.282637 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:19.283071 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:19.283306 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:19.283310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:19.284011 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:19.284264 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:19.284281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.284982 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.284983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:19.285800 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:19.285802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.286127 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.286128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:19.286215 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:19.286601 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:19.286608 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:19.287881 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:19.289133 (+ 1252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:19.289138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:19.289139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:19.289388 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:19.289541 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:19.289541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:19.289542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:19.289626 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:19.289789 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:19.289793 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:19.289885 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 04:48:19.289954 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 04:48:19.290053 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260928 04:48:19.290617 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39397
I20260928 04:48:19.290704 903 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:19.290855 903 heartbeater.cc:511] Master 127.0.98.62:39397 requested a full tablet report, sending...
I20260928 04:48:19.291242 444 ts_manager.cc:194] Registered new tserver with Master: 9f61e353b00a400d9d3c1e0033397456 (127.0.98.3:39283)
I20260928 04:48:19.291603 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56525
I20260928 04:48:19.295389 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570899, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570899, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:48:19.316642 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:19.309857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.0.98.1:44451)
0928 04:48:19.310034 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:19.310038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:19.310047 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:19.310071 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:19.310072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:19.310085 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:19.310147 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:19.310633 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.311296 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.311946 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.312234 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.312281 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:19.314294 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:19.314300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:19.314300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:19.314307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:19.315768 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:19.316024 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:19.316026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:19.316027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:19.316083 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:19.316207 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:19.316207 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:19.316207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:19.316310 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:19.316394 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:19.316446 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:48:19.316499 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
W20260928 04:48:19.317004 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51338
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:48:19.336936 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:19.330503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.0.98.1:44451)
0928 04:48:19.330559 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:19.330561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:19.331005 (+ 444us) server_negotiation.cc:407] Connection header received
0928 04:48:19.331056 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:19.331056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:19.331066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:19.331113 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:19.331635 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.332066 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.332739 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:19.332919 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:19.332971 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:19.335093 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:19.335094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:19.335095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:19.335102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:19.336187 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:19.336460 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:19.336461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:19.336461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:19.336500 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:19.336624 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:19.336624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:19.336625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:19.336690 (+ 65us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:19.336764 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:19.336821 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 04:48:19.336881 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:48:19.337430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 506
I20260928 04:48:19.342368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 640
I20260928 04:48:19.347848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 774
I20260928 04:48:19.352087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 412
2026-09-28T04:48:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1847 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[917](info): set up 2 sockets
Sep 28 04:48:19 dist-test-slave-jccs krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 04:48:20.347044 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 04:48:20.542208 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.121s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 04:48:20.820629 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.278s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/test-user.keytab.
W20260928 04:48:21.400879 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.580s user 0.000s sys 0.001s
Sep 28 04:48:21 dist-test-slave-jccs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260928 04:48:21.631614 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.211s user 0.000s sys 0.003s
I20260928 04:48:21.631929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41129
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:21.711228 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:21.711414 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:21.711436 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:21.712487 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:21.712518 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:21.712531 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:21.712543 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:21.712555 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:21.713855 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:21.714031 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:21.714179 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:21.718484 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:21.718483 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:21.718679 930 server_base.cc:1034] running on GCE node
W20260928 04:48:21.718573 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:21.718957 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:21.719252 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:21.720389 930 hybrid_clock.cc:648] HybridClock initialized: now 1790570901720373 us; error 35 us; skew 500 ppm
Sep 28 04:48:21 dist-test-slave-jccs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:21.725670 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:21.726084 930 webserver.cc:460] Webserver started at http://127.0.98.62:43057/ using document root <none> and password file <none>
I20260928 04:48:21.726281 930 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:21.726322 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:21.726454 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:21.727355 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "8258efbe03fd4e0d85ed330c6334cb32"
format_stamp: "Formatted at 2026-09-28 04:48:21 on dist-test-slave-jccs"
server_key: "5f2c8e14830c739228de0ca90166880c"
server_key_iv: "5f6c7e6acd09b962b036cc76233eea11"
server_key_version: "encryptionkey@0"
I20260928 04:48:21.727659 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "8258efbe03fd4e0d85ed330c6334cb32"
format_stamp: "Formatted at 2026-09-28 04:48:21 on dist-test-slave-jccs"
server_key: "5f2c8e14830c739228de0ca90166880c"
server_key_iv: "5f6c7e6acd09b962b036cc76233eea11"
server_key_version: "encryptionkey@0"
I20260928 04:48:21.728812 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:48:21.729504 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:21.729696 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:21.729775 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "8258efbe03fd4e0d85ed330c6334cb32"
format_stamp: "Formatted at 2026-09-28 04:48:21 on dist-test-slave-jccs"
server_key: "5f2c8e14830c739228de0ca90166880c"
server_key_iv: "5f6c7e6acd09b962b036cc76233eea11"
server_key_version: "encryptionkey@0"
I20260928 04:48:21.729833 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:21.741165 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:21.742961 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:21.743105 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:21.746829 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41129
I20260928 04:48:21.746856 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41129 every 8 connection(s)
I20260928 04:48:21.747165 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:21.747903 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:21.750026 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32: Bootstrap starting.
I20260928 04:48:21.750567 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:21.750870 998 log.cc:824] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:21.751448 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32: No bootstrap required, opened a new log
I20260928 04:48:21.752950 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258efbe03fd4e0d85ed330c6334cb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260928 04:48:21.753082 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:21.753134 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8258efbe03fd4e0d85ed330c6334cb32, State: Initialized, Role: FOLLOWER
I20260928 04:48:21.753321 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258efbe03fd4e0d85ed330c6334cb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260928 04:48:21.753398 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:21.753455 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:21.753499 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:21.754000 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258efbe03fd4e0d85ed330c6334cb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260928 04:48:21.754112 998 leader_election.cc:302] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8258efbe03fd4e0d85ed330c6334cb32; no voters:
I20260928 04:48:21.754276 998 leader_election.cc:288] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:21.754311 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:21.754509 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:21.754630 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [term 1 LEADER]: Becoming Leader. State: Replica: 8258efbe03fd4e0d85ed330c6334cb32, State: Running, Role: LEADER
I20260928 04:48:21.754721 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258efbe03fd4e0d85ed330c6334cb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } }
I20260928 04:48:21.755070 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8258efbe03fd4e0d85ed330c6334cb32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258efbe03fd4e0d85ed330c6334cb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } } }
I20260928 04:48:21.755174 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:21.755084 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8258efbe03fd4e0d85ed330c6334cb32. Latest consensus state: current_term: 1 leader_uuid: "8258efbe03fd4e0d85ed330c6334cb32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8258efbe03fd4e0d85ed330c6334cb32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41129 } } }
I20260928 04:48:21.755384 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:21.755657 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:21.756115 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:21.756440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 930
I20260928 04:48:21.756515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:21.756664 392 external_mini_cluster.cc:1428] Setting key 7506a43ea92659b802f426832b4ca226
I20260928 04:48:21.761890 1010 catalog_manager.cc:1347] Generated new cluster ID: 6248d3d637c84a8f99fed895a311e99b
I20260928 04:48:21.761941 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:48:21 dist-test-slave-jccs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:21.771185 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:21.757024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.0.98.62:41129)
0928 04:48:21.761282 (+ 4258us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:21.761289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:21.761313 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:48:21.761395 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:21.761400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:21.761437 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:21.761541 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:21.762297 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:21.763659 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:21.764687 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:21.765272 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:21.765355 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:21.767896 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:21.767899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:21.767900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:21.767910 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:21.769785 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:21.770116 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:21.770119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:21.770119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:21.770210 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:21.770382 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:21.770383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:21.770383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:21.770637 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:21.770764 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:21.770875 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 04:48:21.770997 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4215,"thread_start_us":78,"threads_started":1}
I20260928 04:48:21.778236 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:21.778726 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:21.782140 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8258efbe03fd4e0d85ed330c6334cb32: Generated new TSK 0
I20260928 04:48:21.782311 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:48:21.808586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:21.886186 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:21.886375 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:21.886410 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:21.887539 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:21.887576 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:21.887601 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:21.888969 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:21.889230 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:21.889446 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:21.893852 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:21.893884 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:21.893977 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:21.894160 1024 server_base.cc:1034] running on GCE node
I20260928 04:48:21.894433 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:21.894680 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:21.895835 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790570901895819 us; error 28 us; skew 500 ppm
Sep 28 04:48:21 dist-test-slave-jccs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:21.898128 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:48:21.898571 1024 webserver.cc:460] Webserver started at http://127.0.98.1:43631/ using document root <none> and password file <none>
I20260928 04:48:21.898777 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:21.898838 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:21.898968 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:21.899938 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ee755dadb3f4eb9b457e07feb604851"
format_stamp: "Formatted at 2026-09-28 04:48:21 on dist-test-slave-jccs"
server_key: "84ade71c94528870e0be96819e80deb2"
server_key_iv: "b928cbae8622e60b0183a7abab465605"
server_key_version: "encryptionkey@0"
I20260928 04:48:21.900282 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ee755dadb3f4eb9b457e07feb604851"
format_stamp: "Formatted at 2026-09-28 04:48:21 on dist-test-slave-jccs"
server_key: "84ade71c94528870e0be96819e80deb2"
server_key_iv: "b928cbae8622e60b0183a7abab465605"
server_key_version: "encryptionkey@0"
I20260928 04:48:21.901618 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:48:21.902281 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:21.902457 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:21.902545 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "4ee755dadb3f4eb9b457e07feb604851"
format_stamp: "Formatted at 2026-09-28 04:48:21 on dist-test-slave-jccs"
server_key: "84ade71c94528870e0be96819e80deb2"
server_key_iv: "b928cbae8622e60b0183a7abab465605"
server_key_version: "encryptionkey@0"
I20260928 04:48:21.902619 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:21.920717 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:21.922422 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:21.922590 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:21.922796 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:21.923069 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:21.923115 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:21.923153 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:21.923187 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:21.928858 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46635
I20260928 04:48:21.928889 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46635 every 8 connection(s)
I20260928 04:48:21.929235 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:48:21.932811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 1024
I20260928 04:48:21.932890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:21.933070 392 external_mini_cluster.cc:1428] Setting key ae87cd36be78a25aca94bcabb4aaf498
Sep 28 04:48:21 dist-test-slave-jccs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:21.938660 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:21.929926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37405 (local address 127.0.98.62:41129)
0928 04:48:21.929983 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:21.929985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:21.930149 (+ 164us) server_negotiation.cc:407] Connection header received
0928 04:48:21.930635 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:21.930636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:21.930646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:21.930687 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:21.931493 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:21.931984 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:21.933096 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:21.933434 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:21.933484 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:21.935722 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:21.935723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:21.935724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:21.935730 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:21.937165 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:21.937724 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:21.937724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:21.937725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:21.937787 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:21.938046 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:21.938047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:21.938047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:21.938147 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:21.938194 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:21.938534 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 04:48:21.938601 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:48:21.938956 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:21.929902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41129 (local address 127.0.98.1:37405)
0928 04:48:21.930039 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:21.930077 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:21.930547 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:21.930739 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:21.930742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:21.931250 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:21.931421 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:21.931425 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:21.932077 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:21.932077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:21.932995 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:21.932997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:21.933491 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:21.933492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:21.933620 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:21.934045 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:21.934054 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:21.935634 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:21.937308 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:21.937314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:21.937317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:21.937651 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:21.937845 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:21.937846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:21.937847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:21.937983 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:21.938208 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:21.938215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:21.938377 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 04:48:21.938504 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 04:48:21.938692 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260928 04:48:21.939455 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41129
I20260928 04:48:21.939539 1153 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:21.939723 1153 heartbeater.cc:511] Master 127.0.98.62:41129 requested a full tablet report, sending...
I20260928 04:48:21.940192 962 ts_manager.cc:194] Registered new tserver with Master: 4ee755dadb3f4eb9b457e07feb604851 (127.0.98.1:46635)
I20260928 04:48:21.940755 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37405
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:48:21.956314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:22.031978 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:22.032193 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:22.032231 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:22.033356 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:22.033407 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:22.033450 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:22.034725 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:22.034993 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:22.035202 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:22.039765 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:22.039773 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:22.039839 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:22.039839 1158 server_base.cc:1034] running on GCE node
I20260928 04:48:22.040134 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:22.040355 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:22.041498 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790570902041478 us; error 33 us; skew 500 ppm
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:22.043699 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:48:22.044090 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43463/ using document root <none> and password file <none>
I20260928 04:48:22.044291 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:22.044349 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:22.044468 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:22.045483 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "dd2d6609513546c897543adced172aa6"
format_stamp: "Formatted at 2026-09-28 04:48:22 on dist-test-slave-jccs"
server_key: "445b7977c36cb16a05ea59a41143b2da"
server_key_iv: "bb615010b76be1800ea4747fa8c35e37"
server_key_version: "encryptionkey@0"
I20260928 04:48:22.045812 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dd2d6609513546c897543adced172aa6"
format_stamp: "Formatted at 2026-09-28 04:48:22 on dist-test-slave-jccs"
server_key: "445b7977c36cb16a05ea59a41143b2da"
server_key_iv: "bb615010b76be1800ea4747fa8c35e37"
server_key_version: "encryptionkey@0"
I20260928 04:48:22.046981 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:48:22.047581 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:22.047747 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:22.047827 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "dd2d6609513546c897543adced172aa6"
format_stamp: "Formatted at 2026-09-28 04:48:22 on dist-test-slave-jccs"
server_key: "445b7977c36cb16a05ea59a41143b2da"
server_key_iv: "bb615010b76be1800ea4747fa8c35e37"
server_key_version: "encryptionkey@0"
I20260928 04:48:22.047894 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:22.072471 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:22.074082 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:22.074227 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:22.074427 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:22.074751 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:22.074801 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:22.074842 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:22.074872 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:22.080294 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32875
I20260928 04:48:22.080340 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32875 every 8 connection(s)
I20260928 04:48:22.080598 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:22.088567 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.081210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56293 (local address 127.0.98.62:41129)
0928 04:48:22.081271 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.081274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.081476 (+ 202us) server_negotiation.cc:407] Connection header received
0928 04:48:22.081916 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.081916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.081925 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.081971 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:22.082826 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.083311 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.084140 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.084304 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.084348 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.086364 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:22.086365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:22.086366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:22.086372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:22.087503 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.087866 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.087867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.087867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.087912 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.088098 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.088099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.088099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.088190 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:22.088237 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.088441 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.088504 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:48:22.088749 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.081195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41129 (local address 127.0.98.2:56293)
0928 04:48:22.081354 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:22.081390 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:22.081842 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:22.082052 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:22.082055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:22.082577 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:22.082753 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.082757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.083378 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.083378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.084062 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.084063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.084362 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.084363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.084467 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.084927 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:22.084940 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:22.086287 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:22.087571 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:22.087576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:22.087577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:22.087804 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:22.087964 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:22.087964 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:22.087964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:22.088051 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:22.088240 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:22.088244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:22.088342 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 04:48:22.088420 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:48:22.088527 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260928 04:48:22.089207 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41129
I20260928 04:48:22.089299 1287 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:22.089461 1287 heartbeater.cc:511] Master 127.0.98.62:41129 requested a full tablet report, sending...
I20260928 04:48:22.089805 962 ts_manager.cc:194] Registered new tserver with Master: dd2d6609513546c897543adced172aa6 (127.0.98.2:32875)
I20260928 04:48:22.090214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 1158
I20260928 04:48:22.090229 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56293
I20260928 04:48:22.090303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:22.090466 392 external_mini_cluster.cc:1428] Setting key 6e71535de9469b402fc0738e3b6998f0
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:48:22.112608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:22.187757 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:22.187964 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:22.188004 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:22.189147 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:22.189229 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:22.189284 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:22.190667 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:22.190907 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:22.191131 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:22.195617 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:22.195693 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:22.195785 1292 server_base.cc:1034] running on GCE node
W20260928 04:48:22.195706 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:22.196118 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:22.196370 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:22.197499 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790570902197481 us; error 34 us; skew 500 ppm
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:22.199828 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:48:22.200165 1292 webserver.cc:460] Webserver started at http://127.0.98.3:37817/ using document root <none> and password file <none>
I20260928 04:48:22.200383 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:22.200426 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:22.200567 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:22.201594 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "45c98bd7c81043c497744403dc54158b"
format_stamp: "Formatted at 2026-09-28 04:48:22 on dist-test-slave-jccs"
server_key: "b7ad8f617e1fdde4800316073e952c88"
server_key_iv: "5c2de9982305c1a993dd35260d9f1ee4"
server_key_version: "encryptionkey@0"
I20260928 04:48:22.201922 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "45c98bd7c81043c497744403dc54158b"
format_stamp: "Formatted at 2026-09-28 04:48:22 on dist-test-slave-jccs"
server_key: "b7ad8f617e1fdde4800316073e952c88"
server_key_iv: "5c2de9982305c1a993dd35260d9f1ee4"
server_key_version: "encryptionkey@0"
I20260928 04:48:22.203152 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:22.203800 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:22.203965 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:22.204046 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "45c98bd7c81043c497744403dc54158b"
format_stamp: "Formatted at 2026-09-28 04:48:22 on dist-test-slave-jccs"
server_key: "b7ad8f617e1fdde4800316073e952c88"
server_key_iv: "5c2de9982305c1a993dd35260d9f1ee4"
server_key_version: "encryptionkey@0"
I20260928 04:48:22.204110 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:22.223652 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:22.225237 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:22.225387 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:22.225580 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:22.225843 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:22.225888 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:22.225925 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:22.225948 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:22.231269 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45321
I20260928 04:48:22.231344 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45321 every 8 connection(s)
I20260928 04:48:22.231632 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:48:22.236819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 1292
I20260928 04:48:22.236917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:22.237073 392 external_mini_cluster.cc:1428] Setting key 9d87a54b5435f7ceaa293c2d14bf06a2
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:22.240193 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.232302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36653 (local address 127.0.98.62:41129)
0928 04:48:22.232350 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.232352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.232544 (+ 192us) server_negotiation.cc:407] Connection header received
0928 04:48:22.233102 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.233103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.233115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.233188 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:22.234050 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.234529 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.235467 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.235674 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.235752 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.238012 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:22.238013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:22.238013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:22.238024 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:22.239151 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.239484 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.239485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.239485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.239530 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.239737 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.239738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.239738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.239829 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:22.239887 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.240084 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.240136 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:48:22.240415 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.232288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41129 (local address 127.0.98.3:36653)
0928 04:48:22.232437 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:22.232469 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:22.233006 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:22.233263 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:22.233267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:22.233782 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:22.233951 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.233956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.234610 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.234611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.235383 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.235385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.235760 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.235762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.235879 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.236366 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:22.236374 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:22.237921 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:22.239216 (+ 1295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:22.239221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:22.239223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:22.239423 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:22.239581 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:22.239582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:22.239582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:22.239677 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:22.239886 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:22.239889 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:22.239983 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 04:48:22.240074 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 04:48:22.240179 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260928 04:48:22.240823 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41129
I20260928 04:48:22.240908 1421 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:22.241061 1421 heartbeater.cc:511] Master 127.0.98.62:41129 requested a full tablet report, sending...
I20260928 04:48:22.241413 962 ts_manager.cc:194] Registered new tserver with Master: 45c98bd7c81043c497744403dc54158b (127.0.98.3:45321)
I20260928 04:48:22.241761 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36653
I20260928 04:48:22.248739 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:48:22.255963 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.250338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.0.98.62:41129)
0928 04:48:22.250393 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.250395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.250484 (+ 89us) server_negotiation.cc:407] Connection header received
0928 04:48:22.250523 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.250524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.250531 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.250571 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:22.251048 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.251536 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.252180 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.252360 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.252406 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.253576 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:22.253577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:22.253577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:22.253582 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:22.254886 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.255270 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.255270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.255271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.255316 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.255465 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.255465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.255466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.255634 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:22.255739 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.255828 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.255883 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:48:22.257970 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:22.259431 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:22.265398 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.261417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.0.98.1:46635)
0928 04:48:22.261770 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.261773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.261781 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:22.261809 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.261810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.261821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.261905 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:22.262050 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.262891 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.264500 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.264971 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.265048 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.265101 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.265276 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.265320 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":70,"threads_started":1}
I20260928 04:48:22.265412 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.261277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46635 (local address 127.0.0.1:52710)
0928 04:48:22.261493 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:22.261504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:22.261565 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:22.261903 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:22.261904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:22.261906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:22.261955 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.261957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.262995 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.262996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.264424 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.264425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.265059 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.265060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.265147 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.265157 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 04:48:22.265258 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 04:48:22.265319 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":22,"thread_start_us":42,"threads_started":1}
I20260928 04:48:22.265560 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.261511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.0.98.2:32875)
0928 04:48:22.261672 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.261676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.262053 (+ 377us) server_negotiation.cc:407] Connection header received
0928 04:48:22.262132 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.262134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.262148 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.262226 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:22.262412 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.263399 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.264281 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.264991 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.265062 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.265116 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.265449 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.265488 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260928 04:48:22.265626 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.261581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.0.98.3:45321)
0928 04:48:22.261742 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.261745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.261752 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:48:22.261779 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.261780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.261792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.261869 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:22.262023 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.263127 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.264330 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.264991 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.265440 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.265493 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.265515 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.265546 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260928 04:48:22.265416 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.261277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45321 (local address 127.0.0.1:58102)
0928 04:48:22.261413 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:22.261446 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:22.261508 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:22.261887 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:22.261890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:22.261892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:22.261951 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.261953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.263226 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.263227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.264266 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.264268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.265111 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.265112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.265211 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.265214 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:22.265247 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 04:48:22.265308 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":1,"thread_start_us":47,"threads_started":1}
I20260928 04:48:22.265415 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.261302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32875 (local address 127.0.0.1:43726)
0928 04:48:22.261985 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:22.261996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:22.262056 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:22.262301 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:22.262302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:22.262304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:22.262341 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.262342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.263491 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.263492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.264198 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.264200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.265141 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.265142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.265211 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.265216 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:48:22.265269 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 04:48:22.265304 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":47,"threads_started":1}
I20260928 04:48:22.266407 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 78bf2f208b5347b792128d665f45a9df (DEFAULT_TABLE table=test-table [id=4d3a2747b752448ca72c2dfe445a25e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:22.266371 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 78bf2f208b5347b792128d665f45a9df (DEFAULT_TABLE table=test-table [id=4d3a2747b752448ca72c2dfe445a25e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:22.266511 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 78bf2f208b5347b792128d665f45a9df (DEFAULT_TABLE table=test-table [id=4d3a2747b752448ca72c2dfe445a25e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:22.266691 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bf2f208b5347b792128d665f45a9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:22.266691 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bf2f208b5347b792128d665f45a9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:22.266855 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bf2f208b5347b792128d665f45a9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:22.269528 1435 tablet_bootstrap.cc:492] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6: Bootstrap starting.
I20260928 04:48:22.270087 1435 tablet_bootstrap.cc:654] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:22.270278 1436 tablet_bootstrap.cc:492] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851: Bootstrap starting.
I20260928 04:48:22.270645 1435 log.cc:824] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:22.270928 1436 tablet_bootstrap.cc:654] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:22.271374 1435 tablet_bootstrap.cc:492] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6: No bootstrap required, opened a new log
I20260928 04:48:22.271445 1435 ts_tablet_manager.cc:1397] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:22.271461 1436 log.cc:824] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:22.271713 1437 tablet_bootstrap.cc:492] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b: Bootstrap starting.
I20260928 04:48:22.272125 1436 tablet_bootstrap.cc:492] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851: No bootstrap required, opened a new log
I20260928 04:48:22.272193 1436 ts_tablet_manager.cc:1397] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:22.272293 1437 tablet_bootstrap.cc:654] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:22.272817 1437 log.cc:824] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:22.273573 1437 tablet_bootstrap.cc:492] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b: No bootstrap required, opened a new log
I20260928 04:48:22.273487 1435 raft_consensus.cc:348] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c98bd7c81043c497744403dc54158b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "4ee755dadb3f4eb9b457e07feb604851" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46635 } } peers { permanent_uuid: "dd2d6609513546c897543adced172aa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260928 04:48:22.273638 1437 ts_tablet_manager.cc:1397] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 04:48:22.273658 1435 raft_consensus.cc:374] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:22.273693 1435 raft_consensus.cc:729] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2d6609513546c897543adced172aa6, State: Initialized, Role: FOLLOWER
I20260928 04:48:22.273810 1435 consensus_queue.cc:260] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c98bd7c81043c497744403dc54158b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "4ee755dadb3f4eb9b457e07feb604851" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46635 } } peers { permanent_uuid: "dd2d6609513546c897543adced172aa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260928 04:48:22.274030 1436 raft_consensus.cc:348] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c98bd7c81043c497744403dc54158b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "4ee755dadb3f4eb9b457e07feb604851" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46635 } } peers { permanent_uuid: "dd2d6609513546c897543adced172aa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260928 04:48:22.274174 1436 raft_consensus.cc:374] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:22.274231 1436 raft_consensus.cc:729] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee755dadb3f4eb9b457e07feb604851, State: Initialized, Role: FOLLOWER
I20260928 04:48:22.274341 1436 consensus_queue.cc:260] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c98bd7c81043c497744403dc54158b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "4ee755dadb3f4eb9b457e07feb604851" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46635 } } peers { permanent_uuid: "dd2d6609513546c897543adced172aa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260928 04:48:22.274505 1435 ts_tablet_manager.cc:1428] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:48:22.274547 1287 heartbeater.cc:503] Master 127.0.98.62:41129 was elected leader, sending a full tablet report...
I20260928 04:48:22.274583 1436 ts_tablet_manager.cc:1428] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:48:22.274698 1153 heartbeater.cc:503] Master 127.0.98.62:41129 was elected leader, sending a full tablet report...
I20260928 04:48:22.275580 1437 raft_consensus.cc:348] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c98bd7c81043c497744403dc54158b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "4ee755dadb3f4eb9b457e07feb604851" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46635 } } peers { permanent_uuid: "dd2d6609513546c897543adced172aa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260928 04:48:22.275727 1437 raft_consensus.cc:374] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:22.275784 1437 raft_consensus.cc:729] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c98bd7c81043c497744403dc54158b, State: Initialized, Role: FOLLOWER
I20260928 04:48:22.275878 1437 consensus_queue.cc:260] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c98bd7c81043c497744403dc54158b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "4ee755dadb3f4eb9b457e07feb604851" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46635 } } peers { permanent_uuid: "dd2d6609513546c897543adced172aa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260928 04:48:22.276119 1437 ts_tablet_manager.cc:1428] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:48:22.276165 1421 heartbeater.cc:503] Master 127.0.98.62:41129 was elected leader, sending a full tablet report...
W20260928 04:48:22.331173 1288 tablet.cc:2369] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:48:22.429955 1154 tablet.cc:2369] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:48:22.478648 1441 raft_consensus.cc:482] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:22.478844 1441 raft_consensus.cc:504] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c98bd7c81043c497744403dc54158b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "4ee755dadb3f4eb9b457e07feb604851" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46635 } } peers { permanent_uuid: "dd2d6609513546c897543adced172aa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260928 04:48:22.479261 1441 leader_election.cc:288] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45c98bd7c81043c497744403dc54158b (127.0.98.3:45321), 4ee755dadb3f4eb9b457e07feb604851 (127.0.98.1:46635)
I20260928 04:48:22.482082 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.479503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38029 (local address 127.0.98.1:46635)
0928 04:48:22.479548 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.479551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.479561 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:22.479596 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.479597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.479609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.479694 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:22.479862 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.480349 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.481353 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.481716 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.481790 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.481800 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.481939 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.481994 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:48:22.482079 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.479422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46635 (local address 127.0.98.2:38029)
0928 04:48:22.479470 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:22.479483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:22.479531 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:22.479701 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:22.479702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:22.479704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:22.479772 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.479774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.480430 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.480431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.481267 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.481269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.481798 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.481799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.481873 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.481876 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:22.481935 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:48:22.481992 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":47}
I20260928 04:48:22.482163 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.479545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39743 (local address 127.0.98.3:45321)
0928 04:48:22.479586 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.479588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.479702 (+ 114us) server_negotiation.cc:407] Connection header received
0928 04:48:22.479770 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.479771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.479782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.479850 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:22.480004 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.480463 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.481437 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.481799 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.481872 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.481881 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.482070 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.482103 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:48:22.482216 1422 tablet.cc:2369] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:48:22.482242 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.479434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45321 (local address 127.0.98.2:39743)
0928 04:48:22.479633 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:22.479641 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:22.479708 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:22.479865 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:22.479865 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:22.479867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:22.479938 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.479940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.480539 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.480540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.481359 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:22.481361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.481885 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.481886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:22.481953 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.481956 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:22.482041 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 04:48:22.482090 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":61,"thread_start_us":89,"threads_started":1}
I20260928 04:48:22.482582 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bf2f208b5347b792128d665f45a9df" candidate_uuid: "dd2d6609513546c897543adced172aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c98bd7c81043c497744403dc54158b" is_pre_election: true
I20260928 04:48:22.482592 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bf2f208b5347b792128d665f45a9df" candidate_uuid: "dd2d6609513546c897543adced172aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee755dadb3f4eb9b457e07feb604851" is_pre_election: true
I20260928 04:48:22.482695 1375 raft_consensus.cc:2454] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd2d6609513546c897543adced172aa6 in term 0.
I20260928 04:48:22.482734 1107 raft_consensus.cc:2454] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd2d6609513546c897543adced172aa6 in term 0.
I20260928 04:48:22.482928 1174 leader_election.cc:302] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45c98bd7c81043c497744403dc54158b, dd2d6609513546c897543adced172aa6; no voters:
I20260928 04:48:22.483068 1441 raft_consensus.cc:2781] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:22.483121 1441 raft_consensus.cc:482] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:22.483181 1441 raft_consensus.cc:3037] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:22.483899 1441 raft_consensus.cc:504] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c98bd7c81043c497744403dc54158b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "4ee755dadb3f4eb9b457e07feb604851" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46635 } } peers { permanent_uuid: "dd2d6609513546c897543adced172aa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260928 04:48:22.484036 1441 leader_election.cc:288] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [CANDIDATE]: Term 1 election: Requested vote from peers 45c98bd7c81043c497744403dc54158b (127.0.98.3:45321), 4ee755dadb3f4eb9b457e07feb604851 (127.0.98.1:46635)
I20260928 04:48:22.484174 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bf2f208b5347b792128d665f45a9df" candidate_uuid: "dd2d6609513546c897543adced172aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c98bd7c81043c497744403dc54158b"
I20260928 04:48:22.484184 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bf2f208b5347b792128d665f45a9df" candidate_uuid: "dd2d6609513546c897543adced172aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee755dadb3f4eb9b457e07feb604851"
I20260928 04:48:22.484248 1375 raft_consensus.cc:3037] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:22.484248 1107 raft_consensus.cc:3037] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:22.485090 1375 raft_consensus.cc:2454] T 78bf2f208b5347b792128d665f45a9df P 45c98bd7c81043c497744403dc54158b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd2d6609513546c897543adced172aa6 in term 1.
I20260928 04:48:22.485090 1107 raft_consensus.cc:2454] T 78bf2f208b5347b792128d665f45a9df P 4ee755dadb3f4eb9b457e07feb604851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd2d6609513546c897543adced172aa6 in term 1.
I20260928 04:48:22.485293 1176 leader_election.cc:302] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee755dadb3f4eb9b457e07feb604851, dd2d6609513546c897543adced172aa6; no voters:
I20260928 04:48:22.485388 1441 raft_consensus.cc:2781] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:22.485518 1441 raft_consensus.cc:686] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [term 1 LEADER]: Becoming Leader. State: Replica: dd2d6609513546c897543adced172aa6, State: Running, Role: LEADER
I20260928 04:48:22.485607 1441 consensus_queue.cc:237] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c98bd7c81043c497744403dc54158b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "4ee755dadb3f4eb9b457e07feb604851" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46635 } } peers { permanent_uuid: "dd2d6609513546c897543adced172aa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260928 04:48:22.486342 962 catalog_manager.cc:5495] T 78bf2f208b5347b792128d665f45a9df P dd2d6609513546c897543adced172aa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd2d6609513546c897543adced172aa6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "dd2d6609513546c897543adced172aa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45c98bd7c81043c497744403dc54158b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee755dadb3f4eb9b457e07feb604851" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd2d6609513546c897543adced172aa6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } health_report { overall_health: HEALTHY } } }
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:48:22.549707 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.542663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.0.98.1:46635)
0928 04:48:22.542722 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.542724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.542882 (+ 158us) server_negotiation.cc:407] Connection header received
0928 04:48:22.542928 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.542929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.542939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.542989 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:22.543481 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.543949 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.544588 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.544761 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.544810 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.547009 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:22.547020 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:22.547020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:22.547027 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:22.548662 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.548936 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.548939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.548939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.548998 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.549143 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.549143 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.549144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.549334 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:22.549497 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.549536 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.549595 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570902, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570902, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:48:22.571050 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.564721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.0.98.1:46635)
0928 04:48:22.564793 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.564795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.564828 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:48:22.564884 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.564885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.564894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.564941 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:22.565478 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.565943 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.566610 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.566789 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.566836 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.568901 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:22.568902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:22.568902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:22.568907 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:22.570093 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.570384 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.570385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.570385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.570429 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.570575 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.570575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.570576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.570692 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:22.570766 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.570884 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.570961 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:48:22.591991 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:22.585640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.0.98.1:46635)
0928 04:48:22.585720 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:22.585723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:22.585797 (+ 74us) server_negotiation.cc:407] Connection header received
0928 04:48:22.585891 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:22.585892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:22.585901 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:22.585962 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:22.586470 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.586932 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.587641 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:22.587833 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:22.587902 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:22.590002 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:22.590003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:22.590003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:22.590010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:22.591132 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.591418 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.591419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.591419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.591464 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:22.591616 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:22.591616 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:22.591617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:22.591697 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:22.591773 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:22.591874 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 04:48:22.591928 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 04:48:22.592430 1067 tablet_service.cc:2908] Rejecting scan request for tablet 78bf2f208b5347b792128d665f45a9df: Uninitialized: clean time has not yet been initialized
I20260928 04:48:22.593340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 1024
I20260928 04:48:22.598284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 1158
I20260928 04:48:22.602730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 1292
I20260928 04:48:22.606860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 930
2026-09-28T04:48:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3255 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[1459](info): set up 2 sockets
Sep 28 04:48:22 dist-test-slave-jccs krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260928 04:48:23.636649 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:23 dist-test-slave-jccs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:23.747391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:23.824060 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:23.824288 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:23.824325 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:23.825451 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:23.825511 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 04:48:23.825533 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:23.825554 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:23.825579 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:23.825600 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:23.827023 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32889
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:23.827281 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:23.827483 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:23.832053 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:23.832118 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:23.832050 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:23.832214 1472 server_base.cc:1034] running on GCE node
I20260928 04:48:23.832513 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:23.832782 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:23.833936 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790570903833919 us; error 28 us; skew 500 ppm
Sep 28 04:48:23 dist-test-slave-jccs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:23.836146 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:23.836517 1472 webserver.cc:460] Webserver started at http://127.0.98.62:36919/ using document root <none> and password file <none>
I20260928 04:48:23.836681 1472 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:23.836720 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:23.836802 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:23.837812 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "fce7ded222894ccea254f1b78141f881"
format_stamp: "Formatted at 2026-09-28 04:48:23 on dist-test-slave-jccs"
server_key: "df5f90c30218934789340b8c46a4c30d"
server_key_iv: "c8944f1a9a7ba2a921a369ae8208d64a"
server_key_version: "encryptionkey@0"
I20260928 04:48:23.838094 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "fce7ded222894ccea254f1b78141f881"
format_stamp: "Formatted at 2026-09-28 04:48:23 on dist-test-slave-jccs"
server_key: "df5f90c30218934789340b8c46a4c30d"
server_key_iv: "c8944f1a9a7ba2a921a369ae8208d64a"
server_key_version: "encryptionkey@0"
I20260928 04:48:23.839299 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:23.839920 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:23.840108 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:23.840174 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "fce7ded222894ccea254f1b78141f881"
format_stamp: "Formatted at 2026-09-28 04:48:23 on dist-test-slave-jccs"
server_key: "df5f90c30218934789340b8c46a4c30d"
server_key_iv: "c8944f1a9a7ba2a921a369ae8208d64a"
server_key_version: "encryptionkey@0"
I20260928 04:48:23.840220 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:23.860517 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:23.862259 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:23.862363 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:23.866835 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32889
I20260928 04:48:23.866899 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32889 every 8 connection(s)
I20260928 04:48:23.867228 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:23.867901 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:23.870126 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881: Bootstrap starting.
I20260928 04:48:23.870679 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:23.870993 1550 log.cc:824] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:23.871240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 1472
I20260928 04:48:23.871328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:23.871477 392 external_mini_cluster.cc:1428] Setting key f575bae92832b96da31e21a66c8ee927
I20260928 04:48:23.871728 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881: No bootstrap required, opened a new log
I20260928 04:48:23.874372 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7ded222894ccea254f1b78141f881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260928 04:48:23.874526 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:23.874555 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce7ded222894ccea254f1b78141f881, State: Initialized, Role: FOLLOWER
I20260928 04:48:23.874660 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7ded222894ccea254f1b78141f881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260928 04:48:23.874722 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:23.874748 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:23.874783 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:23.875370 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7ded222894ccea254f1b78141f881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260928 04:48:23.875451 1550 leader_election.cc:302] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fce7ded222894ccea254f1b78141f881; no voters:
I20260928 04:48:23.875619 1550 leader_election.cc:288] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:23.875641 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:23.875743 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [term 1 LEADER]: Becoming Leader. State: Replica: fce7ded222894ccea254f1b78141f881, State: Running, Role: LEADER
I20260928 04:48:23.875870 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:23.875886 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7ded222894ccea254f1b78141f881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260928 04:48:23.876274 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fce7ded222894ccea254f1b78141f881. Latest consensus state: current_term: 1 leader_uuid: "fce7ded222894ccea254f1b78141f881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7ded222894ccea254f1b78141f881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } } }
I20260928 04:48:23.876356 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:23.876571 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fce7ded222894ccea254f1b78141f881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce7ded222894ccea254f1b78141f881" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } } }
I20260928 04:48:23.876883 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881 [sys.catalog]: This master's current role is: LEADER
W20260928 04:48:23.877761 1571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:48:23.877822 1571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:48:23.877864 1560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:23.878031 1560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 04:48:23 dist-test-slave-jccs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:23.879448 1560 catalog_manager.cc:1347] Generated new cluster ID: a4a17ce8f2164a7ab0ff6601a9869c2c
I20260928 04:48:23.879500 1560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:23.882474 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:23.871947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.0.98.62:32889)
0928 04:48:23.872134 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:23.872143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:23.872177 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:48:23.872948 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:23.872956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:23.873015 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:23.873152 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:23.873740 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:23.875029 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:23.875849 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:23.876387 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:23.876464 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:23.879311 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:23.879314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:23.879314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:23.879329 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:23.880981 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:23.881389 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:23.881393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:23.881393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:23.881474 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:23.881667 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:23.881668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:23.881668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:23.881884 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:23.882065 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:23.882155 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 04:48:23.882302 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260928 04:48:23.889302 1560 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:23.889972 1560 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:23.896281 1560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881: Generated new TSK 0
I20260928 04:48:23.896497 1560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:48:23.927501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:24.005129 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:24.005354 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:24.005379 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:24.006493 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:24.006534 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:24.006554 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:48:24.006608 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:24.008028 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:24.008306 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:24.008548 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:24.013159 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:24.013159 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:24.013200 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:24.013445 1576 server_base.cc:1034] running on GCE node
I20260928 04:48:24.013679 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:24.013865 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:24.015007 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790570904014984 us; error 38 us; skew 500 ppm
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:24.017408 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:48:24.017810 1576 webserver.cc:460] Webserver started at http://127.0.98.1:43573/ using document root <none> and password file <none>
I20260928 04:48:24.018023 1576 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:24.018067 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:24.018224 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:24.019265 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "d4c9c583b66740dfb8261b7caef227c2"
format_stamp: "Formatted at 2026-09-28 04:48:24 on dist-test-slave-jccs"
server_key: "9da397089a4da67e0a632bb17ee57361"
server_key_iv: "6a71be8125363967a3346c9f92e29da4"
server_key_version: "encryptionkey@0"
I20260928 04:48:24.019611 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d4c9c583b66740dfb8261b7caef227c2"
format_stamp: "Formatted at 2026-09-28 04:48:24 on dist-test-slave-jccs"
server_key: "9da397089a4da67e0a632bb17ee57361"
server_key_iv: "6a71be8125363967a3346c9f92e29da4"
server_key_version: "encryptionkey@0"
I20260928 04:48:24.020887 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:48:24.021749 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.021936 1576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:48:24.022022 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "d4c9c583b66740dfb8261b7caef227c2"
format_stamp: "Formatted at 2026-09-28 04:48:24 on dist-test-slave-jccs"
server_key: "9da397089a4da67e0a632bb17ee57361"
server_key_iv: "6a71be8125363967a3346c9f92e29da4"
server_key_version: "encryptionkey@0"
I20260928 04:48:24.022092 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:24.034590 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:24.036258 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:24.036405 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:24.036852 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:24.036886 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.036940 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:24.036978 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:24.043648 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33491
I20260928 04:48:24.043747 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33491 every 8 connection(s)
I20260928 04:48:24.044044 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:48:24.047041 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.037396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48449 (local address 127.0.98.62:32889)
0928 04:48:24.037463 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.037465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.037618 (+ 153us) server_negotiation.cc:407] Connection header received
0928 04:48:24.038081 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.038082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.038096 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.038175 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:24.038981 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.039836 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.041381 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.041724 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.041802 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.044144 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:24.044146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:24.044147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:24.044154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:24.045657 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.046079 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.046080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.046080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.046139 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.046398 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.046399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.046400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.046516 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:24.046590 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.046840 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.046929 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:48:24.047191 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.037356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.1:48449)
0928 04:48:24.037497 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.037534 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.037992 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.038259 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.038264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.038580 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:24.038867 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.038873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.039931 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.039932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.041286 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.041288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.041806 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.041807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.041915 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.042422 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:24.042431 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:24.044051 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:24.045751 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.045757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.045759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.045992 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.046201 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.046202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.046202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.046317 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.046579 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:24.046585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:24.046718 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.046819 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.046972 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260928 04:48:24.047648 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260928 04:48:24.047747 1707 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:24.047911 1707 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260928 04:48:24.048439 1503 ts_manager.cc:194] Registered new tserver with Master: d4c9c583b66740dfb8261b7caef227c2 (127.0.98.1:33491)
I20260928 04:48:24.049263 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48449
I20260928 04:48:24.051950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 1576
I20260928 04:48:24.052052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:24.052232 392 external_mini_cluster.cc:1428] Setting key b789bd22b0678c542049019b54cf594b
I20260928 04:48:24.053862 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.048644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32919 (local address 127.0.98.62:32889)
0928 04:48:24.048681 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.048683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.048803 (+ 120us) server_negotiation.cc:407] Connection header received
0928 04:48:24.048829 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.048830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.048839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.048906 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:24.049149 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.049786 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.050499 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.050666 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.050719 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.051447 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:24.051448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:24.051449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:24.051454 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:24.052801 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.053206 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.053207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.053207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.053263 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.053475 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.053475 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.053475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.053564 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:24.053608 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.053753 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.053807 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:48:24.054001 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.048601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.1:32919)
0928 04:48:24.048733 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.048742 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.048797 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.048909 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.048910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.049050 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:24.049095 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.049096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.049873 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.049875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.050433 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.050435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.050728 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.050729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.050773 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.051110 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:24.051114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:24.051380 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:24.052890 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.052891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.052891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.053106 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.053328 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.053328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.053329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.053407 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.053617 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:24.053618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:24.053663 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.053731 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.053842 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":5,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:48:24.076123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:24.154250 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:24.154440 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:24.154461 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:24.155473 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:24.155505 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:24.155524 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:48:24.155540 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:24.156821 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:24.156978 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:24.157125 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:24.161762 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:24.161809 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:24.161770 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:24.162061 1716 server_base.cc:1034] running on GCE node
I20260928 04:48:24.162245 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:24.162492 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:24.163645 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790570904163636 us; error 35 us; skew 500 ppm
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:24.165928 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:48:24.166345 1716 webserver.cc:460] Webserver started at http://127.0.98.2:44439/ using document root <none> and password file <none>
I20260928 04:48:24.166546 1716 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:24.166604 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:24.166725 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:24.167667 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "d3945f6e0b6842e49fb19fa2e2695f9d"
format_stamp: "Formatted at 2026-09-28 04:48:24 on dist-test-slave-jccs"
server_key: "42d2a7c980d63286cd3d311e7387cfde"
server_key_iv: "0899da22b6bc2022d7b239c0bbfc9f68"
server_key_version: "encryptionkey@0"
I20260928 04:48:24.168026 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d3945f6e0b6842e49fb19fa2e2695f9d"
format_stamp: "Formatted at 2026-09-28 04:48:24 on dist-test-slave-jccs"
server_key: "42d2a7c980d63286cd3d311e7387cfde"
server_key_iv: "0899da22b6bc2022d7b239c0bbfc9f68"
server_key_version: "encryptionkey@0"
I20260928 04:48:24.169291 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:48:24.169963 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.170141 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.170239 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "d3945f6e0b6842e49fb19fa2e2695f9d"
format_stamp: "Formatted at 2026-09-28 04:48:24 on dist-test-slave-jccs"
server_key: "42d2a7c980d63286cd3d311e7387cfde"
server_key_iv: "0899da22b6bc2022d7b239c0bbfc9f68"
server_key_version: "encryptionkey@0"
I20260928 04:48:24.170317 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:24.186867 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:24.188668 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:24.188827 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:24.189383 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:24.189435 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.189477 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:24.189496 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:24.196352 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39191
I20260928 04:48:24.196664 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:48:24.196661 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39191 every 8 connection(s)
I20260928 04:48:24.198436 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.189760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58529 (local address 127.0.98.62:32889)
0928 04:48:24.189805 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.189807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.190059 (+ 252us) server_negotiation.cc:407] Connection header received
0928 04:48:24.190608 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.190610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.190622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.190665 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:24.191310 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.191841 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.192827 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.193030 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.193107 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.195659 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:24.195664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:24.195665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:24.195670 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:24.197165 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.197554 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.197556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.197556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.197609 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.197862 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.197863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.197864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.197969 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:24.198040 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.198280 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.198350 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:48:24.198629 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.189763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.2:58529)
0928 04:48:24.189904 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.189937 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.190503 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.190741 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.190746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.191031 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:24.191204 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.191208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.191940 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.191941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.192722 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.192723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.193116 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.193117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.193267 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.193737 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:24.193750 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:24.195551 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:24.197269 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.197274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.197276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.197485 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.197694 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.197695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.197695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.197797 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.198032 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:24.198037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:24.198162 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.198261 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.198401 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260928 04:48:24.199119 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260928 04:48:24.199213 1847 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:24.199379 1847 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260928 04:48:24.199713 1504 ts_manager.cc:194] Registered new tserver with Master: d3945f6e0b6842e49fb19fa2e2695f9d (127.0.98.2:39191)
I20260928 04:48:24.200091 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58529
I20260928 04:48:24.200474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 1716
I20260928 04:48:24.200557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:24.200752 392 external_mini_cluster.cc:1428] Setting key 68f88de3aafc18ace7171b3459ade5f4
I20260928 04:48:24.205643 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.200053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.2:56617)
0928 04:48:24.200198 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.200210 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.200286 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.200445 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.200446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.200649 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:24.200693 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.200695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.201499 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.201500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.201978 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.201980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.202383 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.202384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.202468 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.202889 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:24.202894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:24.203149 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:24.204755 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.204755 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.204756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.204900 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.205079 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.205079 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.205079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.205129 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.205448 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:24.205449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:24.205477 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.205523 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.205585 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260928 04:48:24.205729 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.200099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56617 (local address 127.0.98.62:32889)
0928 04:48:24.200134 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.200136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.200263 (+ 127us) server_negotiation.cc:407] Connection header received
0928 04:48:24.200377 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.200378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.200387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.200442 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:24.200779 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.201389 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.202059 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.202264 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.202352 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.203211 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:24.203212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:24.203213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:24.203219 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:24.204676 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.204964 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.204965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.204965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.205019 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.205199 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.205200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.205200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.205382 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:24.205460 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.205550 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.205620 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:48:24.223879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:24.300661 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:24.300859 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:24.300882 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:24.301973 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:24.302017 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:24.302037 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:48:24.302055 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:24.303460 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:24.303678 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:24.303843 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:24.308439 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:24.308593 1856 server_base.cc:1034] running on GCE node
W20260928 04:48:24.308450 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:24.308471 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:24.308867 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:24.309121 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:24.310250 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790570904310239 us; error 27 us; skew 500 ppm
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:24.312508 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:48:24.312923 1856 webserver.cc:460] Webserver started at http://127.0.98.3:37675/ using document root <none> and password file <none>
I20260928 04:48:24.313169 1856 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:24.313267 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:24.313429 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:24.314448 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "13c1f012615a4408b20807eb046bd370"
format_stamp: "Formatted at 2026-09-28 04:48:24 on dist-test-slave-jccs"
server_key: "85819d6453affa6f08f499e954c81a77"
server_key_iv: "122a016d9aa9a0f8d565fc9709ae33a2"
server_key_version: "encryptionkey@0"
I20260928 04:48:24.314785 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "13c1f012615a4408b20807eb046bd370"
format_stamp: "Formatted at 2026-09-28 04:48:24 on dist-test-slave-jccs"
server_key: "85819d6453affa6f08f499e954c81a77"
server_key_iv: "122a016d9aa9a0f8d565fc9709ae33a2"
server_key_version: "encryptionkey@0"
I20260928 04:48:24.316046 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:24.316684 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.316855 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:24.316946 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "13c1f012615a4408b20807eb046bd370"
format_stamp: "Formatted at 2026-09-28 04:48:24 on dist-test-slave-jccs"
server_key: "85819d6453affa6f08f499e954c81a77"
server_key_iv: "122a016d9aa9a0f8d565fc9709ae33a2"
server_key_version: "encryptionkey@0"
I20260928 04:48:24.317030 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:24.326759 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:24.328588 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:24.328740 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:24.329325 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:24.329381 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.329422 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:24.329452 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:24.336760 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46869
I20260928 04:48:24.337096 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:48:24.337069 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46869 every 8 connection(s)
I20260928 04:48:24.338059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 1856
I20260928 04:48:24.338131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:24.338322 392 external_mini_cluster.cc:1428] Setting key afabb74e7985d04522deb3c37ee2305d
I20260928 04:48:24.339594 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.329698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37509 (local address 127.0.98.62:32889)
0928 04:48:24.329762 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.329764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.330029 (+ 265us) server_negotiation.cc:407] Connection header received
0928 04:48:24.330775 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.330776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.330786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.330869 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:24.331790 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.332280 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.333996 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.334231 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.334321 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.336485 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:24.336486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:24.336487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:24.336494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:24.338162 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.338601 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.338603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.338603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.338668 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.338965 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.338966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.338966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.339086 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:24.339163 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.339419 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.339504 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:48:24.339784 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.329696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.3:37509)
0928 04:48:24.329878 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.329927 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.330668 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.330956 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.330961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.331363 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:24.331703 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.331711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.332371 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.332372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.333897 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.333900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.334341 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.334342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.334460 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.335018 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:24.335026 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:24.336382 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:24.338254 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.338259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.338261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.338536 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.338729 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.338730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.338730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.338889 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.339154 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:24.339160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:24.339296 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.339398 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.339554 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260928 04:48:24.340224 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260928 04:48:24.340299 1987 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:24.340517 1987 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260928 04:48:24.340852 1504 ts_manager.cc:194] Registered new tserver with Master: 13c1f012615a4408b20807eb046bd370 (127.0.98.3:46869)
I20260928 04:48:24.341367 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37509
I20260928 04:48:24.346500 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.341375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49105 (local address 127.0.98.62:32889)
0928 04:48:24.341414 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.341416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.341524 (+ 108us) server_negotiation.cc:407] Connection header received
0928 04:48:24.341593 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.341594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.341603 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.341648 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:24.341952 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.342424 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.343152 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.343351 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.343428 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.344264 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:24.344265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:24.344265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:24.344272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:24.345662 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.346039 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.346040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.346040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.346083 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.346196 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.346196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.346197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.346276 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:24.346317 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.346394 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.346446 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:48:24.346516 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.341328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.3:49105)
0928 04:48:24.341462 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.341474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.341538 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.341651 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.341651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.341818 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:24.341873 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.341874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.342524 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.342525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.343060 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.343061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.343430 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.343431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.343503 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.343916 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:24.343920 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:24.344183 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:24.345747 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.345749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.345749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.345980 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.346120 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.346121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.346121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.346165 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.346308 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:24.346308 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:24.346335 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.346378 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.346444 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260928 04:48:24.349810 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570904, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570904, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:24.379976 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.372620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.0.98.62:32889)
0928 04:48:24.372703 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.372706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.372716 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:24.372749 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.372750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.372761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.372825 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:24.373669 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.374517 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.375478 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.375702 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.375755 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.378059 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:24.378060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:24.378060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:24.378066 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:24.379211 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.379489 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.379490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.379490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.379537 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.379651 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.379651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.379652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.379736 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:24.379799 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.379859 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.379908 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:48:24.381376 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:24.382920 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:24.388804 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.384988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.0.98.3:46869)
0928 04:48:24.385309 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.385314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.385328 (+ 14us) server_negotiation.cc:407] Connection header received
0928 04:48:24.385358 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.385359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.385371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.385462 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:24.385592 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.386431 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.387785 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.388376 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.388454 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.388529 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.388669 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.388706 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":75,"threads_started":1}
I20260928 04:48:24.388875 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.384908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33491 (local address 127.0.0.1:40958)
0928 04:48:24.385167 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.385202 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.385253 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.385682 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.385682 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.385684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:24.385714 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.385715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.386969 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.386970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.387764 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.387766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.388608 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.388609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.388670 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.388674 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.388704 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.388743 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":51,"threads_started":1}
I20260928 04:48:24.388810 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.384906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46869 (local address 127.0.0.1:58124)
0928 04:48:24.385060 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.385098 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.385164 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.385456 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.385459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.385461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:24.385523 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.385525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.386519 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.386520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.387640 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.387643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.388453 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.388454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.388547 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.388560 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.388652 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.388703 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260928 04:48:24.389106 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.384991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.0.98.1:33491)
0928 04:48:24.385515 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.385519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.385527 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:24.385564 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.385567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.385582 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.385777 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:24.385831 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.386867 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.387851 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.388509 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.388850 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.388930 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.388964 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.389002 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260928 04:48:24.389149 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.385093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39191 (local address 127.0.0.1:48176)
0928 04:48:24.385414 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.385423 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.385471 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.385648 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.385649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.385650 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:24.385705 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.385707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.386932 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.386932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.387978 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.387980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.388923 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.388924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.388986 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.388990 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.389041 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.389077 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":31,"threads_started":1}
I20260928 04:48:24.389431 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.385247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.0.98.2:39191)
0928 04:48:24.385401 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.385405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.385461 (+ 56us) server_negotiation.cc:407] Connection header received
0928 04:48:24.385551 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.385553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.385568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.385651 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:24.385784 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.386848 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.388056 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.388756 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.388816 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.388889 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.389291 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.389338 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260928 04:48:24.389843 1921 tablet_service.cc:1467] Processing CreateTablet for tablet abca06aff9954e8f80f635e87ac34136 (DEFAULT_TABLE table=test-table [id=f9957315fede442cafa336a439a5b43b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:24.390030 1641 tablet_service.cc:1467] Processing CreateTablet for tablet abca06aff9954e8f80f635e87ac34136 (DEFAULT_TABLE table=test-table [id=f9957315fede442cafa336a439a5b43b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:24.390030 1781 tablet_service.cc:1467] Processing CreateTablet for tablet abca06aff9954e8f80f635e87ac34136 (DEFAULT_TABLE table=test-table [id=f9957315fede442cafa336a439a5b43b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:24.390196 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abca06aff9954e8f80f635e87ac34136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:24.390360 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abca06aff9954e8f80f635e87ac34136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:24.390360 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abca06aff9954e8f80f635e87ac34136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:24.392731 2007 tablet_bootstrap.cc:492] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: Bootstrap starting.
I20260928 04:48:24.393344 2007 tablet_bootstrap.cc:654] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:24.393641 2007 log.cc:824] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:24.393905 2008 tablet_bootstrap.cc:492] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: Bootstrap starting.
I20260928 04:48:24.394398 2007 tablet_bootstrap.cc:492] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: No bootstrap required, opened a new log
I20260928 04:48:24.394455 2007 ts_tablet_manager.cc:1397] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:24.394502 2008 tablet_bootstrap.cc:654] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:24.394744 2008 log.cc:824] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:24.395057 2009 tablet_bootstrap.cc:492] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: Bootstrap starting.
I20260928 04:48:24.395358 2008 tablet_bootstrap.cc:492] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: No bootstrap required, opened a new log
I20260928 04:48:24.395428 2008 ts_tablet_manager.cc:1397] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:24.395926 2007 raft_consensus.cc:348] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } }
I20260928 04:48:24.396054 2007 raft_consensus.cc:374] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:24.396075 2007 raft_consensus.cc:729] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c1f012615a4408b20807eb046bd370, State: Initialized, Role: FOLLOWER
I20260928 04:48:24.396198 2007 consensus_queue.cc:260] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } }
I20260928 04:48:24.396448 2007 ts_tablet_manager.cc:1428] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:48:24.396484 1987 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
I20260928 04:48:24.397459 2008 raft_consensus.cc:348] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } }
I20260928 04:48:24.397660 2008 raft_consensus.cc:374] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:24.397714 2008 raft_consensus.cc:729] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4c9c583b66740dfb8261b7caef227c2, State: Initialized, Role: FOLLOWER
I20260928 04:48:24.397831 2008 consensus_queue.cc:260] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } }
I20260928 04:48:24.398084 1707 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
I20260928 04:48:24.398069 2008 ts_tablet_manager.cc:1428] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:48:24.398643 2009 tablet_bootstrap.cc:654] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:24.398957 2009 log.cc:824] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:24.399569 2009 tablet_bootstrap.cc:492] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: No bootstrap required, opened a new log
I20260928 04:48:24.399618 2009 ts_tablet_manager.cc:1397] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 04:48:24.401021 2009 raft_consensus.cc:348] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } }
I20260928 04:48:24.401122 2009 raft_consensus.cc:374] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:24.401140 2009 raft_consensus.cc:729] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3945f6e0b6842e49fb19fa2e2695f9d, State: Initialized, Role: FOLLOWER
I20260928 04:48:24.401281 2009 consensus_queue.cc:260] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } }
I20260928 04:48:24.401567 1847 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
I20260928 04:48:24.401553 2009 ts_tablet_manager.cc:1428] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 04:48:24.447206 1848 tablet.cc:2369] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:48:24.544742 1708 tablet.cc:2369] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:48:24.587720 1988 tablet.cc:2369] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:48:24.643641 2013 raft_consensus.cc:482] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:24.643764 2013 raft_consensus.cc:504] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } }
I20260928 04:48:24.644171 2013 leader_election.cc:288] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13c1f012615a4408b20807eb046bd370 (127.0.98.3:46869), d3945f6e0b6842e49fb19fa2e2695f9d (127.0.98.2:39191)
I20260928 04:48:24.647192 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.644529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34023 (local address 127.0.98.3:46869)
0928 04:48:24.644592 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.644594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.644660 (+ 66us) server_negotiation.cc:407] Connection header received
0928 04:48:24.644730 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.644731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.644742 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.644798 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:24.644958 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.645462 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.646493 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.646860 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.646940 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.646950 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.647100 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.647133 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:48:24.647192 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.644370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39191 (local address 127.0.98.1:55883)
0928 04:48:24.644508 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.644519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.644628 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.644766 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.644767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.644770 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:24.644845 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.644847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.645561 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.645562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.646383 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.646385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.646928 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.646929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.646994 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.646998 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.647055 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.647112 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":33,"thread_start_us":70,"threads_started":1}
I20260928 04:48:24.647213 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.644440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55883 (local address 127.0.98.2:39191)
0928 04:48:24.644510 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.644512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.644631 (+ 119us) server_negotiation.cc:407] Connection header received
0928 04:48:24.644694 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.644695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.644709 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.644756 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:24.644912 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.645468 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.646466 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.646851 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.646925 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.646936 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.647063 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.647106 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:48:24.647393 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.644370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46869 (local address 127.0.98.1:34023)
0928 04:48:24.644575 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.644585 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.644674 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.644811 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.644812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.644814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:24.644866 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.644868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.645536 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.645538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.646387 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.646390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.646951 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.646952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.647014 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.647018 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.647087 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.647124 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":42,"thread_start_us":69,"threads_started":1}
I20260928 04:48:24.647677 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abca06aff9954e8f80f635e87ac34136" candidate_uuid: "d4c9c583b66740dfb8261b7caef227c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c1f012615a4408b20807eb046bd370" is_pre_election: true
I20260928 04:48:24.647675 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abca06aff9954e8f80f635e87ac34136" candidate_uuid: "d4c9c583b66740dfb8261b7caef227c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" is_pre_election: true
I20260928 04:48:24.647815 1800 raft_consensus.cc:2454] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4c9c583b66740dfb8261b7caef227c2 in term 0.
I20260928 04:48:24.647822 1941 raft_consensus.cc:2454] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4c9c583b66740dfb8261b7caef227c2 in term 0.
I20260928 04:48:24.648039 1594 leader_election.cc:302] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c1f012615a4408b20807eb046bd370, d4c9c583b66740dfb8261b7caef227c2; no voters:
I20260928 04:48:24.648188 2013 raft_consensus.cc:2781] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:24.648280 2013 raft_consensus.cc:482] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:24.648306 2013 raft_consensus.cc:3037] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:24.648975 2013 raft_consensus.cc:504] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } }
I20260928 04:48:24.649124 2013 leader_election.cc:288] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [CANDIDATE]: Term 1 election: Requested vote from peers 13c1f012615a4408b20807eb046bd370 (127.0.98.3:46869), d3945f6e0b6842e49fb19fa2e2695f9d (127.0.98.2:39191)
I20260928 04:48:24.649297 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abca06aff9954e8f80f635e87ac34136" candidate_uuid: "d4c9c583b66740dfb8261b7caef227c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c1f012615a4408b20807eb046bd370"
I20260928 04:48:24.649386 1941 raft_consensus.cc:3037] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:24.650081 1941 raft_consensus.cc:2454] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4c9c583b66740dfb8261b7caef227c2 in term 1.
I20260928 04:48:24.650197 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abca06aff9954e8f80f635e87ac34136" candidate_uuid: "d4c9c583b66740dfb8261b7caef227c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d"
I20260928 04:48:24.650228 1594 leader_election.cc:302] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c1f012615a4408b20807eb046bd370, d4c9c583b66740dfb8261b7caef227c2; no voters:
I20260928 04:48:24.650272 1800 raft_consensus.cc:3037] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:24.650962 1800 raft_consensus.cc:2454] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4c9c583b66740dfb8261b7caef227c2 in term 1.
I20260928 04:48:24.651068 2013 raft_consensus.cc:2781] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:24.651206 2013 raft_consensus.cc:686] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 1 LEADER]: Becoming Leader. State: Replica: d4c9c583b66740dfb8261b7caef227c2, State: Running, Role: LEADER
I20260928 04:48:24.651319 2013 consensus_queue.cc:237] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } }
I20260928 04:48:24.652076 1504 catalog_manager.cc:5495] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4c9c583b66740dfb8261b7caef227c2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d4c9c583b66740dfb8261b7caef227c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } health_report { overall_health: UNKNOWN } } }
I20260928 04:48:24.675112 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.672683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.0.98.1:33491)
0928 04:48:24.672738 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.672740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.672750 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:24.672803 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.672804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.672822 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.672896 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:24.673020 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.673558 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.674344 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.674526 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.674590 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.674790 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:24.674885 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:24.674942 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.674995 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.675011 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:48:24.677048 1800 raft_consensus.cc:1261] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d [term 1 FOLLOWER]: Refusing update from remote peer d4c9c583b66740dfb8261b7caef227c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:24.677242 1941 raft_consensus.cc:1261] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370 [term 1 FOLLOWER]: Refusing update from remote peer d4c9c583b66740dfb8261b7caef227c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:24.677385 2013 consensus_queue.cc:1035] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:24.677549 2018 consensus_queue.cc:1035] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:24.679698 2021 mvcc.cc:205] Tried to move back new op lower bound from 7334178425554157568 to 7334178425452388352. Current Snapshot: MvccSnapshot[applied={T|T < 7334178425554157568}]
I20260928 04:48:24.679809 2022 mvcc.cc:205] Tried to move back new op lower bound from 7334178425554157568 to 7334178425452388352. Current Snapshot: MvccSnapshot[applied={T|T < 7334178425554157568}]
I20260928 04:48:24.683482 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:24.683568 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:24.684413 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881: Sending DeleteTablet for 3 replicas of tablet abca06aff9954e8f80f635e87ac34136
I20260928 04:48:24.684744 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet abca06aff9954e8f80f635e87ac34136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58124
I20260928 04:48:24.684755 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet abca06aff9954e8f80f635e87ac34136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48176
I20260928 04:48:24.684831 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet abca06aff9954e8f80f635e87ac34136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40958
I20260928 04:48:24.684973 2031 tablet_replica.cc:330] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: stopping tablet replica
I20260928 04:48:24.685063 2031 raft_consensus.cc:2229] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:24.685190 2031 raft_consensus.cc:2258] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:24.685320 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:24.685491 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:24.685493 2032 tablet_replica.cc:330] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: stopping tablet replica
I20260928 04:48:24.685544 2030 tablet_replica.cc:330] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: stopping tablet replica
I20260928 04:48:24.685585 2032 raft_consensus.cc:2229] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:24.685617 2030 raft_consensus.cc:2229] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:24.685695 2032 raft_consensus.cc:2258] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:24.685717 2030 raft_consensus.cc:2258] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:24.685738 2031 ts_tablet_manager.cc:1905] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:24.686182 2030 ts_tablet_manager.cc:1905] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:24.686977 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 58d96af851174913b2866ae5937cdea8 (DEFAULT_TABLE table=test-table [id=34b2d1436a9b420c9bf8ab1891f9030a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:24.687036 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 58d96af851174913b2866ae5937cdea8 (DEFAULT_TABLE table=test-table [id=34b2d1436a9b420c9bf8ab1891f9030a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:24.687088 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d96af851174913b2866ae5937cdea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:24.687140 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d96af851174913b2866ae5937cdea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:24.688041 2030 ts_tablet_manager.cc:1918] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:24.688105 2030 log.cc:1196] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/abca06aff9954e8f80f635e87ac34136
I20260928 04:48:24.688385 2030 ts_tablet_manager.cc:1939] T abca06aff9954e8f80f635e87ac34136 P 13c1f012615a4408b20807eb046bd370: Deleting consensus metadata
I20260928 04:48:24.688755 2008 tablet_bootstrap.cc:492] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2: Bootstrap starting.
I20260928 04:48:24.688781 1490 catalog_manager.cc:4841] TS 13c1f012615a4408b20807eb046bd370 (127.0.98.3:46869): tablet abca06aff9954e8f80f635e87ac34136 (table test-table [id=f9957315fede442cafa336a439a5b43b]) successfully deleted
I20260928 04:48:24.688813 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 58d96af851174913b2866ae5937cdea8 (DEFAULT_TABLE table=test-table [id=34b2d1436a9b420c9bf8ab1891f9030a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:24.688966 2007 tablet_bootstrap.cc:492] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370: Bootstrap starting.
I20260928 04:48:24.689666 2008 tablet_bootstrap.cc:654] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:24.689774 2007 tablet_bootstrap.cc:654] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:24.690780 2008 tablet_bootstrap.cc:492] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2: No bootstrap required, opened a new log
I20260928 04:48:24.690840 2008 ts_tablet_manager.cc:1397] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:48:24.690958 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d96af851174913b2866ae5937cdea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:24.691004 2008 raft_consensus.cc:348] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.691092 2008 raft_consensus.cc:374] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:24.691119 2008 raft_consensus.cc:729] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4c9c583b66740dfb8261b7caef227c2, State: Initialized, Role: FOLLOWER
I20260928 04:48:24.691160 2008 consensus_queue.cc:260] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.691244 2008 ts_tablet_manager.cc:1428] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:24.691998 2009 tablet_bootstrap.cc:492] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d: Bootstrap starting.
I20260928 04:48:24.692193 2007 tablet_bootstrap.cc:492] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370: No bootstrap required, opened a new log
I20260928 04:48:24.692233 2007 ts_tablet_manager.cc:1397] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 04:48:24.692337 2007 raft_consensus.cc:348] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.692380 2007 raft_consensus.cc:374] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:24.692394 2007 raft_consensus.cc:729] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c1f012615a4408b20807eb046bd370, State: Initialized, Role: FOLLOWER
I20260928 04:48:24.692386 2009 tablet_bootstrap.cc:654] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:24.692451 2007 consensus_queue.cc:260] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.692560 2007 ts_tablet_manager.cc:1428] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.693069 2009 tablet_bootstrap.cc:492] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d: No bootstrap required, opened a new log
I20260928 04:48:24.693113 2009 ts_tablet_manager.cc:1397] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:24.693228 2031 ts_tablet_manager.cc:1918] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:24.693257 2009 raft_consensus.cc:348] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.693300 2031 log.cc:1196] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/abca06aff9954e8f80f635e87ac34136
I20260928 04:48:24.693318 2009 raft_consensus.cc:374] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:24.693339 2009 raft_consensus.cc:729] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3945f6e0b6842e49fb19fa2e2695f9d, State: Initialized, Role: FOLLOWER
I20260928 04:48:24.693380 2009 consensus_queue.cc:260] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.693454 2009 ts_tablet_manager.cc:1428] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.693555 2031 ts_tablet_manager.cc:1939] T abca06aff9954e8f80f635e87ac34136 P d3945f6e0b6842e49fb19fa2e2695f9d: Deleting consensus metadata
I20260928 04:48:24.694005 1488 catalog_manager.cc:4841] TS d3945f6e0b6842e49fb19fa2e2695f9d (127.0.98.2:39191): tablet abca06aff9954e8f80f635e87ac34136 (table test-table [id=f9957315fede442cafa336a439a5b43b]) successfully deleted
I20260928 04:48:24.694166 2032 ts_tablet_manager.cc:1905] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:24.695454 2032 ts_tablet_manager.cc:1918] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:24.695510 2032 log.cc:1196] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/abca06aff9954e8f80f635e87ac34136
I20260928 04:48:24.695676 2032 ts_tablet_manager.cc:1939] T abca06aff9954e8f80f635e87ac34136 P d4c9c583b66740dfb8261b7caef227c2: Deleting consensus metadata
I20260928 04:48:24.696257 1488 catalog_manager.cc:4841] TS d4c9c583b66740dfb8261b7caef227c2 (127.0.98.1:33491): tablet abca06aff9954e8f80f635e87ac34136 (table test-table [id=f9957315fede442cafa336a439a5b43b]) successfully deleted
I20260928 04:48:24.739957 2012 raft_consensus.cc:482] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:24.740056 2012 raft_consensus.cc:504] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.740375 2012 leader_election.cc:288] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4c9c583b66740dfb8261b7caef227c2 (127.0.98.1:33491), d3945f6e0b6842e49fb19fa2e2695f9d (127.0.98.2:39191)
I20260928 04:48:24.743880 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.740560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33491 (local address 127.0.98.3:55201)
0928 04:48:24.740627 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.740639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.740684 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.740852 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.740852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.740854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:24.740929 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.740931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.741689 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.741691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.742840 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.742842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.743613 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.743615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.743687 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.743691 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.743755 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.743802 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 04:48:24.743875 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.740671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55201 (local address 127.0.98.1:33491)
0928 04:48:24.740713 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.740715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.740724 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:24.740751 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.740752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.740776 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.740839 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:24.741010 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.741608 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.742933 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.743533 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.743607 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.743618 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.743770 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.743808 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:48:24.744194 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d96af851174913b2866ae5937cdea8" candidate_uuid: "13c1f012615a4408b20807eb046bd370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4c9c583b66740dfb8261b7caef227c2" is_pre_election: true
I20260928 04:48:24.744305 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.740950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60197 (local address 127.0.98.2:39191)
0928 04:48:24.740989 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.740991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.740998 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:48:24.741022 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.741022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.741030 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.741081 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:24.741269 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.741962 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.743181 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.743701 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.743763 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.743773 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.744196 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.744240 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:48:24.744323 1661 raft_consensus.cc:2454] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c1f012615a4408b20807eb046bd370 in term 0.
I20260928 04:48:24.744527 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.740686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39191 (local address 127.0.98.3:60197)
0928 04:48:24.740826 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.740835 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.740875 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.741100 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.741101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.741104 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:24.741202 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.741203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.742053 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.742054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.743101 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.743102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.744017 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.744018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.744099 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.744104 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.744410 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.744452 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
I20260928 04:48:24.744503 1872 leader_election.cc:302] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c1f012615a4408b20807eb046bd370, d4c9c583b66740dfb8261b7caef227c2; no voters:
I20260928 04:48:24.744665 2012 raft_consensus.cc:2781] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:24.744700 2012 raft_consensus.cc:482] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:24.744720 2012 raft_consensus.cc:3037] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:24.744794 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d96af851174913b2866ae5937cdea8" candidate_uuid: "13c1f012615a4408b20807eb046bd370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" is_pre_election: true
I20260928 04:48:24.744880 1800 raft_consensus.cc:2454] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c1f012615a4408b20807eb046bd370 in term 0.
I20260928 04:48:24.745615 2012 raft_consensus.cc:504] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.745740 2012 leader_election.cc:288] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [CANDIDATE]: Term 1 election: Requested vote from peers d4c9c583b66740dfb8261b7caef227c2 (127.0.98.1:33491), d3945f6e0b6842e49fb19fa2e2695f9d (127.0.98.2:39191)
I20260928 04:48:24.745882 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d96af851174913b2866ae5937cdea8" candidate_uuid: "13c1f012615a4408b20807eb046bd370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4c9c583b66740dfb8261b7caef227c2"
I20260928 04:48:24.745930 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d96af851174913b2866ae5937cdea8" candidate_uuid: "13c1f012615a4408b20807eb046bd370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d"
I20260928 04:48:24.745960 1661 raft_consensus.cc:3037] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:24.746007 1800 raft_consensus.cc:3037] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:24.746703 1661 raft_consensus.cc:2454] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c1f012615a4408b20807eb046bd370 in term 1.
I20260928 04:48:24.746721 1800 raft_consensus.cc:2454] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c1f012615a4408b20807eb046bd370 in term 1.
I20260928 04:48:24.746884 1872 leader_election.cc:302] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c1f012615a4408b20807eb046bd370, d3945f6e0b6842e49fb19fa2e2695f9d; no voters:
I20260928 04:48:24.746994 2012 raft_consensus.cc:2781] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:24.747165 2012 raft_consensus.cc:686] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 1 LEADER]: Becoming Leader. State: Replica: 13c1f012615a4408b20807eb046bd370, State: Running, Role: LEADER
I20260928 04:48:24.747249 2012 consensus_queue.cc:237] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.747869 1503 catalog_manager.cc:5495] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13c1f012615a4408b20807eb046bd370 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "13c1f012615a4408b20807eb046bd370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } health_report { overall_health: HEALTHY } } }
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:24.789906 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.782005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.0.98.62:32889)
0928 04:48:24.782077 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.782079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.782224 (+ 145us) server_negotiation.cc:407] Connection header received
0928 04:48:24.782295 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.782296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.782304 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.782367 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:24.783367 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.783872 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.784866 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.785047 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.785113 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.787319 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:24.787321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:24.787321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:24.787327 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:24.788693 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.789064 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.789065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.789065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.789120 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.789343 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.789344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.789345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.789448 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:24.789536 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.789688 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.789795 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":45}
I20260928 04:48:24.789969 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.781935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.0.1:57200)
0928 04:48:24.782131 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.782144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.782227 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.782395 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.782396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.782802 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:24.783266 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.783269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.783958 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.783960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.784757 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.784760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.785133 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.785135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.785221 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.785752 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:24.785757 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:24.787218 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:24.788791 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.788798 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.788799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.788990 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.789220 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:24.789220 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:24.789221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:24.789282 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:24.789529 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:24.789536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:24.789580 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.789664 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.789849 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":85,"thread_start_us":99,"threads_started":1}
I20260928 04:48:24.791821 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57200:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 04:48:24.792016 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:24.793360 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 1faf5f30da784ee4ba3039dc6811a769 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45d6b525f01d42599a0328d499cbd7e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:48:24.793354 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 1faf5f30da784ee4ba3039dc6811a769 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45d6b525f01d42599a0328d499cbd7e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:48:24.793419 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 1faf5f30da784ee4ba3039dc6811a769 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=45d6b525f01d42599a0328d499cbd7e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:48:24.793502 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1faf5f30da784ee4ba3039dc6811a769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:24.793507 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1faf5f30da784ee4ba3039dc6811a769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:24.793555 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1faf5f30da784ee4ba3039dc6811a769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:24.794914 2007 tablet_bootstrap.cc:492] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: Bootstrap starting.
I20260928 04:48:24.794912 2008 tablet_bootstrap.cc:492] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: Bootstrap starting.
I20260928 04:48:24.795298 2007 tablet_bootstrap.cc:654] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:24.795331 2008 tablet_bootstrap.cc:654] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:24.795647 2009 tablet_bootstrap.cc:492] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: Bootstrap starting.
I20260928 04:48:24.795972 2008 tablet_bootstrap.cc:492] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: No bootstrap required, opened a new log
I20260928 04:48:24.796007 2008 ts_tablet_manager.cc:1397] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:24.796015 2009 tablet_bootstrap.cc:654] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:24.796108 2008 raft_consensus.cc:348] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.796152 2008 raft_consensus.cc:374] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:24.796166 2008 raft_consensus.cc:729] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4c9c583b66740dfb8261b7caef227c2, State: Initialized, Role: FOLLOWER
I20260928 04:48:24.796226 2008 consensus_queue.cc:260] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.796324 2008 ts_tablet_manager.cc:1428] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.796406 2013 tablet_replica.cc:439] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } }
I20260928 04:48:24.796505 2013 tablet_replica.cc:442] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:24.796577 2009 tablet_bootstrap.cc:492] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: No bootstrap required, opened a new log
I20260928 04:48:24.796624 2009 ts_tablet_manager.cc:1397] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:24.796767 2009 raft_consensus.cc:348] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.796831 2009 raft_consensus.cc:374] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:24.796830 2007 tablet_bootstrap.cc:492] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: No bootstrap required, opened a new log
I20260928 04:48:24.796854 2009 raft_consensus.cc:729] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3945f6e0b6842e49fb19fa2e2695f9d, State: Initialized, Role: FOLLOWER
I20260928 04:48:24.796871 2007 ts_tablet_manager.cc:1397] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:48:24.796897 2009 consensus_queue.cc:260] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.797008 2009 ts_tablet_manager.cc:1428] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:24.797008 2007 raft_consensus.cc:348] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.797068 2007 raft_consensus.cc:374] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:24.797087 2007 raft_consensus.cc:729] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c1f012615a4408b20807eb046bd370, State: Initialized, Role: FOLLOWER
I20260928 04:48:24.797127 2007 consensus_queue.cc:260] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.797118 2015 tablet_replica.cc:439] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } }
I20260928 04:48:24.797206 2015 tablet_replica.cc:442] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:24.797297 2007 ts_tablet_manager.cc:1428] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:24.798170 2012 tablet_replica.cc:439] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } }
I20260928 04:48:24.798254 2012 tablet_replica.cc:442] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:24.809032 2012 raft_consensus.cc:482] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:24.809109 2012 raft_consensus.cc:504] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.809257 2012 leader_election.cc:288] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4c9c583b66740dfb8261b7caef227c2 (127.0.98.1:33491), d3945f6e0b6842e49fb19fa2e2695f9d (127.0.98.2:39191)
I20260928 04:48:24.809423 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1faf5f30da784ee4ba3039dc6811a769" candidate_uuid: "13c1f012615a4408b20807eb046bd370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4c9c583b66740dfb8261b7caef227c2" is_pre_election: true
I20260928 04:48:24.809514 1661 raft_consensus.cc:2454] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c1f012615a4408b20807eb046bd370 in term 0.
I20260928 04:48:24.809482 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1faf5f30da784ee4ba3039dc6811a769" candidate_uuid: "13c1f012615a4408b20807eb046bd370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" is_pre_election: true
I20260928 04:48:24.809589 1800 raft_consensus.cc:2454] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c1f012615a4408b20807eb046bd370 in term 0.
I20260928 04:48:24.809646 1872 leader_election.cc:302] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c1f012615a4408b20807eb046bd370, d4c9c583b66740dfb8261b7caef227c2; no voters:
I20260928 04:48:24.809721 2012 raft_consensus.cc:2781] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:24.809759 2012 raft_consensus.cc:482] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:24.809778 2012 raft_consensus.cc:3037] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:24.810662 2012 raft_consensus.cc:504] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.810788 2012 leader_election.cc:288] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [CANDIDATE]: Term 1 election: Requested vote from peers d4c9c583b66740dfb8261b7caef227c2 (127.0.98.1:33491), d3945f6e0b6842e49fb19fa2e2695f9d (127.0.98.2:39191)
I20260928 04:48:24.810915 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1faf5f30da784ee4ba3039dc6811a769" candidate_uuid: "13c1f012615a4408b20807eb046bd370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4c9c583b66740dfb8261b7caef227c2"
I20260928 04:48:24.810993 1661 raft_consensus.cc:3037] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:24.810942 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1faf5f30da784ee4ba3039dc6811a769" candidate_uuid: "13c1f012615a4408b20807eb046bd370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d"
I20260928 04:48:24.811029 1800 raft_consensus.cc:3037] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:24.811564 1661 raft_consensus.cc:2454] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c1f012615a4408b20807eb046bd370 in term 1.
I20260928 04:48:24.811568 1800 raft_consensus.cc:2454] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c1f012615a4408b20807eb046bd370 in term 1.
I20260928 04:48:24.811717 1872 leader_election.cc:302] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c1f012615a4408b20807eb046bd370, d4c9c583b66740dfb8261b7caef227c2; no voters:
I20260928 04:48:24.811825 2012 raft_consensus.cc:2781] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:24.811889 2012 raft_consensus.cc:686] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [term 1 LEADER]: Becoming Leader. State: Replica: 13c1f012615a4408b20807eb046bd370, State: Running, Role: LEADER
I20260928 04:48:24.811937 2012 consensus_queue.cc:237] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } }
I20260928 04:48:24.812255 2037 tablet_replica.cc:439] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: TxnStatusTablet state changed. Reason: New leader 13c1f012615a4408b20807eb046bd370. Latest consensus state: current_term: 1 leader_uuid: "13c1f012615a4408b20807eb046bd370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } }
I20260928 04:48:24.812348 2037 tablet_replica.cc:442] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:48:24.812557 2049 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 04:48:24.812534 1503 catalog_manager.cc:5495] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13c1f012615a4408b20807eb046bd370 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "13c1f012615a4408b20807eb046bd370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } health_report { overall_health: HEALTHY } } }
I20260928 04:48:24.812665 2049 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 04:48:24.822714 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.820174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.0.98.3:46869)
0928 04:48:24.820220 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.820222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.820274 (+ 52us) server_negotiation.cc:407] Connection header received
0928 04:48:24.820373 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.820373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.820388 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.820467 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:24.820592 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.821099 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.821876 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.822083 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.822166 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.822377 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:24.822487 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:24.822567 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.822623 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.822643 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:48:24.822734 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.820144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46869 (local address 127.0.0.1:58130)
0928 04:48:24.820227 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.820238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.820293 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.820474 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.820475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.820476 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:48:24.820525 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.820527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.821223 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.821224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.821789 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.821791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.822168 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.822168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.822239 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.822261 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:48:24.822553 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:48:24.822562 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.822618 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.822664 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 04:48:24.823900 1800 raft_consensus.cc:1261] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d [term 1 FOLLOWER]: Refusing update from remote peer 13c1f012615a4408b20807eb046bd370: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:24.823978 1661 raft_consensus.cc:1261] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2 [term 1 FOLLOWER]: Refusing update from remote peer 13c1f012615a4408b20807eb046bd370: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:24.824141 2037 consensus_queue.cc:1035] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:24.824254 2012 consensus_queue.cc:1035] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:24.824811 2015 tablet_replica.cc:439] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: TxnStatusTablet state changed. Reason: New leader 13c1f012615a4408b20807eb046bd370. Latest consensus state: current_term: 1 leader_uuid: "13c1f012615a4408b20807eb046bd370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } }
I20260928 04:48:24.824904 2015 tablet_replica.cc:442] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:24.824894 2013 tablet_replica.cc:439] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: TxnStatusTablet state changed. Reason: New leader 13c1f012615a4408b20807eb046bd370. Latest consensus state: current_term: 1 leader_uuid: "13c1f012615a4408b20807eb046bd370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } }
I20260928 04:48:24.824992 2013 tablet_replica.cc:442] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:24.825611 2012 tablet_replica.cc:439] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13c1f012615a4408b20807eb046bd370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } }
I20260928 04:48:24.825700 2012 tablet_replica.cc:442] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:48:24.826226 2013 tablet_replica.cc:439] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13c1f012615a4408b20807eb046bd370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } }
I20260928 04:48:24.826313 2013 tablet_replica.cc:442] T 1faf5f30da784ee4ba3039dc6811a769 P d4c9c583b66740dfb8261b7caef227c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:24.827004 2012 tablet_replica.cc:439] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13c1f012615a4408b20807eb046bd370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } }
I20260928 04:48:24.827076 2012 tablet_replica.cc:442] T 1faf5f30da784ee4ba3039dc6811a769 P 13c1f012615a4408b20807eb046bd370: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:48:24.827153 2015 tablet_replica.cc:439] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13c1f012615a4408b20807eb046bd370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } } peers { permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 } } peers { permanent_uuid: "13c1f012615a4408b20807eb046bd370" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46869 } } }
I20260928 04:48:24.827224 2015 tablet_replica.cc:442] T 1faf5f30da784ee4ba3039dc6811a769 P d3945f6e0b6842e49fb19fa2e2695f9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:24.829901 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.827798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.0.98.3:46869)
0928 04:48:24.827831 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.827833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.827840 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:48:24.827859 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.827860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.827869 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.827924 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:24.828020 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.828465 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.829224 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.829471 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.829533 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.829681 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:24.829711 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:24.829764 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.829816 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.829832 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:48:24.834439 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.832069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46869 (local address 127.0.98.3:55507)
0928 04:48:24.832136 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:24.832147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:24.832194 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:24.832317 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:24.832318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:24.832318 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:48:24.832367 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.832369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.832957 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.832958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.833675 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:24.833677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.834036 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.834037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:24.834100 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.834111 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:48:24.834284 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:48:24.834286 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:48:24.834333 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 04:48:24.834373 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":3}
I20260928 04:48:24.834438 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.832136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55507 (local address 127.0.98.3:46869)
0928 04:48:24.832167 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.832169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.832188 (+ 19us) server_negotiation.cc:407] Connection header received
0928 04:48:24.832253 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.832254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.832261 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.832314 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:24.832445 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.832862 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.833756 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.833954 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.834034 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.834189 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:24.834217 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:24.834284 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.834336 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.834356 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":17}
I20260928 04:48:24.836920 1800 raft_consensus.cc:1261] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d [term 1 FOLLOWER]: Refusing update from remote peer 13c1f012615a4408b20807eb046bd370: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:24.836935 1661 raft_consensus.cc:1261] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2 [term 1 FOLLOWER]: Refusing update from remote peer 13c1f012615a4408b20807eb046bd370: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:24.837112 2056 consensus_queue.cc:1035] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3945f6e0b6842e49fb19fa2e2695f9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:24.837248 2037 consensus_queue.cc:1035] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4c9c583b66740dfb8261b7caef227c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:24.850771 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:24.850868 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:24.851454 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fce7ded222894ccea254f1b78141f881: Sending DeleteTablet for 3 replicas of tablet 58d96af851174913b2866ae5937cdea8
I20260928 04:48:24.851692 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 58d96af851174913b2866ae5937cdea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58124
I20260928 04:48:24.851732 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 58d96af851174913b2866ae5937cdea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48176
I20260928 04:48:24.851739 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 58d96af851174913b2866ae5937cdea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40958
I20260928 04:48:24.851783 2030 tablet_replica.cc:330] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370: stopping tablet replica
I20260928 04:48:24.851835 2030 raft_consensus.cc:2229] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:24.851837 2032 tablet_replica.cc:330] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2: stopping tablet replica
I20260928 04:48:24.851837 2031 tablet_replica.cc:330] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d: stopping tablet replica
I20260928 04:48:24.851886 2031 raft_consensus.cc:2229] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:24.851892 2032 raft_consensus.cc:2229] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:24.851922 2031 raft_consensus.cc:2258] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:24.851922 2030 raft_consensus.cc:2258] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:24.851945 2032 raft_consensus.cc:2258] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:24.852211 2031 ts_tablet_manager.cc:1905] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:24.852277 2030 ts_tablet_manager.cc:1905] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:24.852284 2032 ts_tablet_manager.cc:1905] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:24.854117 2032 ts_tablet_manager.cc:1918] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:24.854123 2031 ts_tablet_manager.cc:1918] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:24.854180 2031 log.cc:1196] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/58d96af851174913b2866ae5937cdea8
I20260928 04:48:24.854182 2032 log.cc:1196] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/58d96af851174913b2866ae5937cdea8
I20260928 04:48:24.854404 2032 ts_tablet_manager.cc:1939] T 58d96af851174913b2866ae5937cdea8 P d4c9c583b66740dfb8261b7caef227c2: Deleting consensus metadata
I20260928 04:48:24.854420 2031 ts_tablet_manager.cc:1939] T 58d96af851174913b2866ae5937cdea8 P d3945f6e0b6842e49fb19fa2e2695f9d: Deleting consensus metadata
I20260928 04:48:24.854713 1488 catalog_manager.cc:4841] TS d3945f6e0b6842e49fb19fa2e2695f9d (127.0.98.2:39191): tablet 58d96af851174913b2866ae5937cdea8 (table test-table [id=34b2d1436a9b420c9bf8ab1891f9030a]) successfully deleted
I20260928 04:48:24.854746 2030 ts_tablet_manager.cc:1918] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:24.854784 1488 catalog_manager.cc:4841] TS d4c9c583b66740dfb8261b7caef227c2 (127.0.98.1:33491): tablet 58d96af851174913b2866ae5937cdea8 (table test-table [id=34b2d1436a9b420c9bf8ab1891f9030a]) successfully deleted
I20260928 04:48:24.854792 2030 log.cc:1196] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/58d96af851174913b2866ae5937cdea8
I20260928 04:48:24.854986 2030 ts_tablet_manager.cc:1939] T 58d96af851174913b2866ae5937cdea8 P 13c1f012615a4408b20807eb046bd370: Deleting consensus metadata
I20260928 04:48:24.855257 1490 catalog_manager.cc:4841] TS 13c1f012615a4408b20807eb046bd370 (127.0.98.3:46869): tablet 58d96af851174913b2866ae5937cdea8 (table test-table [id=34b2d1436a9b420c9bf8ab1891f9030a]) successfully deleted
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570904, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:48:24.863219 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.854813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.0.98.1:33491)
0928 04:48:24.854853 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.854855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.855161 (+ 306us) server_negotiation.cc:407] Connection header received
0928 04:48:24.855235 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.855235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.855245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.855296 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:24.855871 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.856333 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.857061 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.857329 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.857410 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.860369 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:24.860375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:24.860375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:24.860381 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:24.861971 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.862375 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.862378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.862378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.862466 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.862668 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.862668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.862668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.862793 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:24.862923 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.863029 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.863113 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 04:48:24.863454 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40974
I20260928 04:48:24.870471 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:24.864443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.0.98.62:32889)
0928 04:48:24.864501 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:24.864503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:24.864542 (+ 39us) server_negotiation.cc:407] Connection header received
0928 04:48:24.864619 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:24.864620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:24.864628 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:24.864712 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:24.865339 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.865825 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.866507 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:24.866757 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:24.866817 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:24.867892 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:24.867893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:24.867893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:24.867900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:24.869409 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.869740 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.869741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.869742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.869801 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:24.869995 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:24.869996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:24.869996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:24.870108 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:24.870210 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:24.870301 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 04:48:24.870374 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 04:48:24.870738 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57212
I20260928 04:48:24.871397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 1576
I20260928 04:48:24.878585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 1716
I20260928 04:48:24.884739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 1856
I20260928 04:48:24.889890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 1472
2026-09-28T04:48:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2283 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[2066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[2066](info): set up 2 sockets
Sep 28 04:48:24 dist-test-slave-jccs krb5kdc[2066](info): commencing operation
krb5kdc: starting...
W20260928 04:48:25.916862 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:26.042207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:26.120563 2079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:26.120805 2079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:26.120854 2079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:26.122020 2079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:26.122109 2079 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 04:48:26.122135 2079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:26.122167 2079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:26.122190 2079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:26.122216 2079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:26.123672 2079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39517
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:26.123950 2079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:26.124171 2079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:26.128965 2087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:26.128965 2084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:26.129158 2079 server_base.cc:1034] running on GCE node
W20260928 04:48:26.129015 2085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:26.129805 2079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:26.130096 2079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:26.131263 2079 hybrid_clock.cc:648] HybridClock initialized: now 1790570906131249 us; error 43 us; skew 500 ppm
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:26.133862 2079 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:26.134358 2079 webserver.cc:460] Webserver started at http://127.0.98.62:43243/ using document root <none> and password file <none>
I20260928 04:48:26.134603 2079 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:26.134672 2079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:26.134819 2079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:26.135872 2079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "aa272a547d524242ac0132becf5a7d38"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "8b13ce256dd9057d852b8ea7f7904743"
server_key_iv: "192e3c16e44a687ad3cc43d0c782f6df"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.136235 2079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa272a547d524242ac0132becf5a7d38"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "8b13ce256dd9057d852b8ea7f7904743"
server_key_iv: "192e3c16e44a687ad3cc43d0c782f6df"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.137614 2079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:26.138302 2094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:26.138459 2079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:26.138577 2079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "aa272a547d524242ac0132becf5a7d38"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "8b13ce256dd9057d852b8ea7f7904743"
server_key_iv: "192e3c16e44a687ad3cc43d0c782f6df"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.138648 2079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:26.152940 2079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:26.155354 2079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:26.155510 2079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:26.160084 2079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39517
I20260928 04:48:26.160132 2156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39517 every 8 connection(s)
I20260928 04:48:26.160450 2079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:26.161058 2157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:26.163389 2157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38: Bootstrap starting.
I20260928 04:48:26.163987 2157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:26.164273 2157 log.cc:824] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:26.164947 2157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38: No bootstrap required, opened a new log
I20260928 04:48:26.166736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 2079
I20260928 04:48:26.166846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:26.167042 392 external_mini_cluster.cc:1428] Setting key a139e40f47f32f57af01a48dddba6d69
I20260928 04:48:26.167400 2157 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa272a547d524242ac0132becf5a7d38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } }
I20260928 04:48:26.167591 2157 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:26.167646 2157 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa272a547d524242ac0132becf5a7d38, State: Initialized, Role: FOLLOWER
I20260928 04:48:26.167785 2157 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa272a547d524242ac0132becf5a7d38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } }
I20260928 04:48:26.167870 2157 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:26.167902 2157 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:26.167949 2157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:26.168570 2157 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa272a547d524242ac0132becf5a7d38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } }
I20260928 04:48:26.168659 2157 leader_election.cc:302] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa272a547d524242ac0132becf5a7d38; no voters:
I20260928 04:48:26.168828 2157 leader_election.cc:288] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:26.168861 2162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:26.169010 2162 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [term 1 LEADER]: Becoming Leader. State: Replica: aa272a547d524242ac0132becf5a7d38, State: Running, Role: LEADER
I20260928 04:48:26.169075 2157 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:26.169104 2162 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa272a547d524242ac0132becf5a7d38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } }
I20260928 04:48:26.169471 2163 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa272a547d524242ac0132becf5a7d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa272a547d524242ac0132becf5a7d38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } } }
I20260928 04:48:26.169567 2163 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:26.169456 2162 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa272a547d524242ac0132becf5a7d38. Latest consensus state: current_term: 1 leader_uuid: "aa272a547d524242ac0132becf5a7d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa272a547d524242ac0132becf5a7d38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39517 } } }
I20260928 04:48:26.169655 2162 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:26.170044 2168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:26.170522 2168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:26.173278 2168 catalog_manager.cc:1347] Generated new cluster ID: 34fefce97b584a87a927bcb34ec72a41
I20260928 04:48:26.173336 2168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:26.178368 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.167568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.0.98.62:39517)
0928 04:48:26.167777 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.167786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.167819 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:48:26.167948 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.167956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.168014 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.168176 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:26.168819 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.170371 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.171452 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.171847 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.171925 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.174750 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:26.174754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:26.174754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:26.174766 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:26.176699 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.177103 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.177106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.177107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.177260 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.177498 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.177499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.177499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.177796 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:26.177948 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.178052 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.178180 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
I20260928 04:48:26.186646 2168 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:26.187137 2168 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:26.195904 2168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38: Generated new TSK 0
I20260928 04:48:26.196100 2168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:48:26.225998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39517
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:26.308722 2183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:26.308960 2183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:26.309005 2183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:26.310226 2183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:26.310295 2183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:26.310334 2183 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 04:48:26.310360 2183 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:48:26.310390 2183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:26.311854 2183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:26.312137 2183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:26.312371 2183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:26.317379 2189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:26.317399 2191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:26.317440 2188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:26.317639 2183 server_base.cc:1034] running on GCE node
I20260928 04:48:26.317808 2183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:26.318011 2183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:26.319152 2183 hybrid_clock.cc:648] HybridClock initialized: now 1790570906319135 us; error 32 us; skew 500 ppm
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:26.321791 2183 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:48:26.322216 2183 webserver.cc:460] Webserver started at http://127.0.98.1:45469/ using document root <none> and password file <none>
I20260928 04:48:26.322446 2183 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:26.322492 2183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:26.322675 2183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:26.323704 2183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "62bd621b36ce4624b16e107889eea5f8"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "e32f0783293cf6912044a5dbbee9332a"
server_key_iv: "6b4481f4cdd8d6ff372b59d89e581880"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.324061 2183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "62bd621b36ce4624b16e107889eea5f8"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "e32f0783293cf6912044a5dbbee9332a"
server_key_iv: "6b4481f4cdd8d6ff372b59d89e581880"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.325379 2183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:48:26.326076 2198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:26.326253 2183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:26.326321 2183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "62bd621b36ce4624b16e107889eea5f8"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "e32f0783293cf6912044a5dbbee9332a"
server_key_iv: "6b4481f4cdd8d6ff372b59d89e581880"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.326411 2183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:26.346246 2183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:26.348706 2183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:26.348873 2183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:26.349499 2183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:26.349543 2183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:26.349567 2183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:26.349601 2183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:26.356459 2183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37437
I20260928 04:48:26.356518 2313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37437 every 8 connection(s)
I20260928 04:48:26.356825 2183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:26.360287 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.349982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51831 (local address 127.0.98.62:39517)
0928 04:48:26.350049 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.350051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.350264 (+ 213us) server_negotiation.cc:407] Connection header received
0928 04:48:26.350820 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.350822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.350842 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.350923 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:26.351758 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.352604 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.354322 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.354600 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.354656 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.357464 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:26.357465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:26.357466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:26.357475 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:26.358890 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.359318 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.359319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.359320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.359398 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.359645 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.359646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.359646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.359764 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:26.359839 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.360103 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.360199 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:48:26.360457 2207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.349969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39517 (local address 127.0.98.1:51831)
0928 04:48:26.350145 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.350194 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.350682 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.351015 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.351019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.351381 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:26.351665 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.351671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.352717 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.352718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.354207 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.354211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.354699 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.354701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.354864 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.355471 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:26.355483 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:26.357357 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:26.358987 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.358993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.358995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.359243 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.359464 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.359465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.359465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.359578 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.359830 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:26.359836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:26.359974 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.360081 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.360234 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":113,"threads_started":1}
I20260928 04:48:26.360934 2314 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39517
I20260928 04:48:26.361047 2314 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:26.361287 2314 heartbeater.cc:511] Master 127.0.98.62:39517 requested a full tablet report, sending...
I20260928 04:48:26.361384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 2183
I20260928 04:48:26.361478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:26.361730 392 external_mini_cluster.cc:1428] Setting key c9052da90316dcbb0a6e8ff194c31900
I20260928 04:48:26.361860 2110 ts_manager.cc:194] Registered new tserver with Master: 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1:37437)
I20260928 04:48:26.362772 2110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51831
I20260928 04:48:26.368032 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.362131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45741 (local address 127.0.98.62:39517)
0928 04:48:26.362165 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.362167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.362175 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:26.362198 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.362199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.362208 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.362266 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:26.362607 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.363250 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.364346 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.364521 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.364578 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.365589 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:26.365590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:26.365590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:26.365595 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:26.366969 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.367380 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.367381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.367381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.367426 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.367598 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.367599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.367599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.367698 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:26.367743 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.367922 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.367976 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:48:26.368122 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.361949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39517 (local address 127.0.98.1:45741)
0928 04:48:26.362085 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.362096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.362146 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.362267 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.362268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.362446 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:26.362507 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.362509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.363343 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.363343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.364263 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.364266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.364595 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.364596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.364676 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.365039 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:26.365044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:26.365511 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:26.367054 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.367055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.367055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.367304 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.367470 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.367471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.367472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.367536 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.367758 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:26.367759 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:26.367806 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.367908 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.368013 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":30,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:48:26.386185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39517
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:26.465857 2323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:26.466070 2323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:26.466097 2323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:26.467195 2323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:26.467232 2323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:26.467253 2323 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 04:48:26.467265 2323 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:48:26.467283 2323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:26.468768 2323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:26.468990 2323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:26.469210 2323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:26.474093 2329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:26.474073 2328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:26.474074 2331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:26.474387 2323 server_base.cc:1034] running on GCE node
I20260928 04:48:26.474534 2323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:26.474776 2323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:26.475920 2323 hybrid_clock.cc:648] HybridClock initialized: now 1790570906475901 us; error 34 us; skew 500 ppm
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:26.478369 2323 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:48:26.478843 2323 webserver.cc:460] Webserver started at http://127.0.98.2:38409/ using document root <none> and password file <none>
I20260928 04:48:26.479069 2323 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:26.479115 2323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:26.479272 2323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:26.480350 2323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "14e3fdf376664610b20dca7afec7e249"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "e5a90971f5b16ee9368058707cb15990"
server_key_iv: "5866c273728ea2475eb8a104757000c9"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.480723 2323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14e3fdf376664610b20dca7afec7e249"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "e5a90971f5b16ee9368058707cb15990"
server_key_iv: "5866c273728ea2475eb8a104757000c9"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.482088 2323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:48:26.482844 2338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:26.483012 2323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:26.483117 2323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "14e3fdf376664610b20dca7afec7e249"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "e5a90971f5b16ee9368058707cb15990"
server_key_iv: "5866c273728ea2475eb8a104757000c9"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.483201 2323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:26.493801 2323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:26.495601 2323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:26.495765 2323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:26.496323 2323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:26.496388 2323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:26.496435 2323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:26.496455 2323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:26.503695 2323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43289
I20260928 04:48:26.503793 2453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43289 every 8 connection(s)
I20260928 04:48:26.504098 2323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:48:26.506083 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.496730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38677 (local address 127.0.98.62:39517)
0928 04:48:26.496778 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.496780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.497058 (+ 278us) server_negotiation.cc:407] Connection header received
0928 04:48:26.497589 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.497590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.497604 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.497710 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:26.498417 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.498932 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.499910 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.500225 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.500299 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.503467 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:26.503468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:26.503469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:26.503476 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:26.504977 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.505383 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.505384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.505384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.505433 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.505623 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.505623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.505623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.505720 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:26.505764 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.505965 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.506021 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:48:26.506237 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.496717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39517 (local address 127.0.98.2:38677)
0928 04:48:26.496921 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.496961 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.497494 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.497785 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.497789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.498136 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:26.498321 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.498326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.499025 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.499027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.499818 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.499820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.500321 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.500322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.500492 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.501076 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:26.501088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:26.503370 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:26.505045 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.505049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.505050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.505330 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.505490 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.505490 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.505491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.505581 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.505769 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:26.505772 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:26.505875 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.505946 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.506054 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":130,"threads_started":1}
I20260928 04:48:26.506745 2454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39517
I20260928 04:48:26.506817 2454 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:26.506991 2454 heartbeater.cc:511] Master 127.0.98.62:39517 requested a full tablet report, sending...
I20260928 04:48:26.507354 2110 ts_manager.cc:194] Registered new tserver with Master: 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289)
I20260928 04:48:26.507869 2110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38677
I20260928 04:48:26.510756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 2323
I20260928 04:48:26.510864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:26.511057 392 external_mini_cluster.cc:1428] Setting key cf83235bdf9b44c31caa725a569b73ba
I20260928 04:48:26.513073 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.507773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57381 (local address 127.0.98.62:39517)
0928 04:48:26.507814 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.507816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.507870 (+ 54us) server_negotiation.cc:407] Connection header received
0928 04:48:26.507940 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.507941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.507950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.507996 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:26.508335 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.508802 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.509582 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.509771 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.509843 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.510592 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:26.510593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:26.510593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:26.510600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:26.511974 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.512291 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.512292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.512292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.512347 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.512541 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.512542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.512542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.512673 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:26.512767 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.512920 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.512991 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:48:26.513090 2460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.507700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39517 (local address 127.0.98.2:57381)
0928 04:48:26.507819 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.507829 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.507885 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.508036 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.508036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.508200 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:26.508250 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.508252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.508896 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.508897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.509492 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.509493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.509842 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.509843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.509920 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.510255 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:26.510259 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:26.510506 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:26.512053 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.512054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.512054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.512221 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.512411 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.512412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.512412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.512468 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.512769 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:26.512770 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:26.512813 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.512898 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.512992 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:48:26.535583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39517
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:26.616626 2463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:26.616824 2463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:26.616849 2463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:26.618021 2463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:26.618062 2463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:26.618083 2463 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 04:48:26.618095 2463 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:48:26.618112 2463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:26.619462 2463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:26.619696 2463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:26.619868 2463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:26.624567 2468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:26.624670 2471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:26.624697 2463 server_base.cc:1034] running on GCE node
W20260928 04:48:26.624585 2469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:26.624922 2463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:26.625102 2463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:26.626237 2463 hybrid_clock.cc:648] HybridClock initialized: now 1790570906626221 us; error 32 us; skew 500 ppm
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:26.628507 2463 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:48:26.628895 2463 webserver.cc:460] Webserver started at http://127.0.98.3:42179/ using document root <none> and password file <none>
I20260928 04:48:26.629076 2463 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:26.629117 2463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:26.629284 2463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:26.630400 2463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "d9c93ab5b2791ad37fc479b9bea58ddf"
server_key_iv: "50bb742b0fe3ad14cb820c5e9eb5b680"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.630702 2463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "d9c93ab5b2791ad37fc479b9bea58ddf"
server_key_iv: "50bb742b0fe3ad14cb820c5e9eb5b680"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.631886 2463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:48:26.632527 2478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:26.632710 2463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:26.632817 2463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33"
format_stamp: "Formatted at 2026-09-28 04:48:26 on dist-test-slave-jccs"
server_key: "d9c93ab5b2791ad37fc479b9bea58ddf"
server_key_iv: "50bb742b0fe3ad14cb820c5e9eb5b680"
server_key_version: "encryptionkey@0"
I20260928 04:48:26.632886 2463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:26.662902 2463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:26.665024 2463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:26.665179 2463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:26.665781 2463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:26.665823 2463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:26.665845 2463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:26.665859 2463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:26.672973 2463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42045
I20260928 04:48:26.673048 2593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42045 every 8 connection(s)
I20260928 04:48:26.673409 2463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:48:26.675838 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.666280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42649 (local address 127.0.98.62:39517)
0928 04:48:26.666358 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.666361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.666551 (+ 190us) server_negotiation.cc:407] Connection header received
0928 04:48:26.667036 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.667037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.667051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.667128 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:26.667976 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.668810 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.670460 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.670833 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.670919 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.673251 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:26.673252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:26.673253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:26.673262 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:26.674717 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.675079 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.675079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.675080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.675129 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.675334 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.675334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.675335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.675435 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:26.675484 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.675703 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.675775 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:48:26.676031 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.666237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39517 (local address 127.0.98.3:42649)
0928 04:48:26.666392 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.666434 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.666939 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.667228 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.667234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.667583 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:26.667870 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.667877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.668954 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.668955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.670366 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.670370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.670933 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.670934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.671080 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.671647 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:26.671657 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:26.673121 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:26.674788 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.674793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.674794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.675004 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.675184 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.675185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.675185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.675281 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.675487 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:26.675491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:26.675598 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.675681 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.675799 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260928 04:48:26.676498 2594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39517
I20260928 04:48:26.676575 2594 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:26.676785 2594 heartbeater.cc:511] Master 127.0.98.62:39517 requested a full tablet report, sending...
I20260928 04:48:26.677134 2111 ts_manager.cc:194] Registered new tserver with Master: 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045)
I20260928 04:48:26.677675 2111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42649
I20260928 04:48:26.680434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 2463
I20260928 04:48:26.680531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:26.680711 392 external_mini_cluster.cc:1428] Setting key f3e3109f985330f955ee5393948fa7f5
I20260928 04:48:26.681567 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:48:26.683250 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.677466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39517 (local address 127.0.98.3:50295)
0928 04:48:26.677614 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.677624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.677697 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.677850 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.677851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.678007 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:26.678057 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.678058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.678697 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.678698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.679226 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.679227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.679578 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.679579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.679666 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.680102 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:26.680107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:26.680410 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:26.682319 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.682320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.682320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.682506 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.682724 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:26.682725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:26.682725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:26.682775 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:26.683022 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:26.683022 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:26.683054 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.683105 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.683180 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260928 04:48:26.683346 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.677520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50295 (local address 127.0.98.62:39517)
0928 04:48:26.677560 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.677563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.677683 (+ 120us) server_negotiation.cc:407] Connection header received
0928 04:48:26.677754 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.677755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.677766 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.677813 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:26.678141 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.678601 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.679306 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.679494 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.679576 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.680515 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:26.680516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:26.680517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:26.680523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:26.682230 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.682580 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.682581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.682581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.682657 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.682827 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.682828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.682828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.682965 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:26.683027 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.683129 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.683209 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:48:26 dist-test-slave-jccs krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:26.737713 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.727104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.0.98.62:39517)
0928 04:48:26.727173 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.727175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.727187 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:48:26.727270 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.727271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.727287 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.727358 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:26.728007 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.728570 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.729597 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.729871 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.729966 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.735278 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:26.735280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:26.735280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:26.735289 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:26.736581 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.736914 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.736915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.736916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.736975 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:26.737194 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:26.737195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:26.737195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:26.737308 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:26.737418 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.737523 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.737607 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:48:26.739462 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:26.741200 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:26.747704 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.743578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42045 (local address 127.0.0.1:43372)
0928 04:48:26.743732 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.743770 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.743855 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.744676 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.744677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.744679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:26.744707 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.744708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.745658 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.745659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.746700 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.746702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.747388 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.747389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.747467 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.747482 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.747568 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.747617 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":66,"thread_start_us":54,"threads_started":1}
I20260928 04:48:26.747711 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.743741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.0.98.3:42045)
0928 04:48:26.744510 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.744514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.744524 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:26.744560 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.744562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.744576 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.744778 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:26.744829 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.745569 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.746782 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.747287 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.747378 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.747438 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.747586 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.747622 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":71,"threads_started":1}
I20260928 04:48:26.748034 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.743578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37437 (local address 127.0.0.1:41032)
0928 04:48:26.743833 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.743842 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.743883 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.744224 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.744227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.744230 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:26.744301 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.744303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.745604 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.745605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.746856 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.746858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.747767 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.747768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.747888 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.747892 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.747922 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.747960 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":4,"thread_start_us":74,"threads_started":1}
I20260928 04:48:26.748449 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.743901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.0.98.1:37437)
0928 04:48:26.744070 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.744075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.744091 (+ 16us) server_negotiation.cc:407] Connection header received
0928 04:48:26.744121 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.744122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.744134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.744237 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:26.744371 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.745505 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.746951 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.747672 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.748210 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.748277 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.748304 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.748340 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260928 04:48:26.748564 2528 tablet_service.cc:1467] Processing CreateTablet for tablet d64fdab4852946e2b058c4c04ba291d7 (DEFAULT_TABLE table=test-table [id=d90c59e22f884000aaec496e65484b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:26.748930 2528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64fdab4852946e2b058c4c04ba291d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:26.749053 2248 tablet_service.cc:1467] Processing CreateTablet for tablet d64fdab4852946e2b058c4c04ba291d7 (DEFAULT_TABLE table=test-table [id=d90c59e22f884000aaec496e65484b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:26.749404 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.743853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.0.98.2:43289)
0928 04:48:26.744038 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.744043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.744054 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:48:26.744129 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.744522 (+ 393us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.744546 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.744637 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:26.744763 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.745869 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.747397 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.748116 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.748194 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.748277 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.749270 (+ 993us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.749310 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260928 04:48:26.748637 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.743596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43289 (local address 127.0.0.1:52600)
0928 04:48:26.743959 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.743966 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.744018 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.744650 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.744651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.744653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:26.744702 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.744704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.745965 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.745966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.747307 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.747310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.748313 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.748314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.748382 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.748387 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.748539 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.748578 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":89,"threads_started":1}
I20260928 04:48:26.749536 2248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64fdab4852946e2b058c4c04ba291d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:26.751549 2617 tablet_bootstrap.cc:492] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Bootstrap starting.
I20260928 04:48:26.752354 2618 tablet_bootstrap.cc:492] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: Bootstrap starting.
I20260928 04:48:26.752452 2617 tablet_bootstrap.cc:654] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:26.752794 2617 log.cc:824] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:26.752947 2618 tablet_bootstrap.cc:654] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:26.753315 2618 log.cc:824] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:26.753403 2388 tablet_service.cc:1467] Processing CreateTablet for tablet d64fdab4852946e2b058c4c04ba291d7 (DEFAULT_TABLE table=test-table [id=d90c59e22f884000aaec496e65484b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:26.753618 2617 tablet_bootstrap.cc:492] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: No bootstrap required, opened a new log
I20260928 04:48:26.753674 2617 ts_tablet_manager.cc:1397] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:48:26.753711 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64fdab4852946e2b058c4c04ba291d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:26.753963 2618 tablet_bootstrap.cc:492] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: No bootstrap required, opened a new log
I20260928 04:48:26.754024 2618 ts_tablet_manager.cc:1397] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:26.755491 2618 raft_consensus.cc:348] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 04:48:26.755611 2618 raft_consensus.cc:374] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:26.755631 2618 raft_consensus.cc:729] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62bd621b36ce4624b16e107889eea5f8, State: Initialized, Role: FOLLOWER
I20260928 04:48:26.755753 2618 consensus_queue.cc:260] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 04:48:26.755688 2617 raft_consensus.cc:348] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 04:48:26.755865 2617 raft_consensus.cc:374] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:26.755909 2617 raft_consensus.cc:729] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79bfbd01e2ac4ceda2b9b220df7c3f33, State: Initialized, Role: FOLLOWER
I20260928 04:48:26.756037 2617 consensus_queue.cc:260] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 04:48:26.756059 2314 heartbeater.cc:503] Master 127.0.98.62:39517 was elected leader, sending a full tablet report...
I20260928 04:48:26.756321 2594 heartbeater.cc:503] Master 127.0.98.62:39517 was elected leader, sending a full tablet report...
I20260928 04:48:26.756052 2618 ts_tablet_manager.cc:1428] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:48:26.756558 2617 ts_tablet_manager.cc:1428] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:48:26.757284 2623 tablet_bootstrap.cc:492] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: Bootstrap starting.
I20260928 04:48:26.757835 2623 tablet_bootstrap.cc:654] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:26.758136 2623 log.cc:824] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:26.758800 2623 tablet_bootstrap.cc:492] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: No bootstrap required, opened a new log
I20260928 04:48:26.758852 2623 ts_tablet_manager.cc:1397] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:26.760272 2623 raft_consensus.cc:348] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 04:48:26.760394 2623 raft_consensus.cc:374] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:26.760416 2623 raft_consensus.cc:729] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e3fdf376664610b20dca7afec7e249, State: Initialized, Role: FOLLOWER
I20260928 04:48:26.760509 2623 consensus_queue.cc:260] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 04:48:26.760797 2454 heartbeater.cc:503] Master 127.0.98.62:39517 was elected leader, sending a full tablet report...
I20260928 04:48:26.760799 2623 ts_tablet_manager.cc:1428] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 04:48:26.857530 2315 tablet.cc:2369] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:48:26.924049 2595 tablet.cc:2369] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:48:26.954648 2621 raft_consensus.cc:482] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:26.954773 2621 raft_consensus.cc:504] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 04:48:26.955147 2621 leader_election.cc:288] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045), 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289)
I20260928 04:48:26.958349 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.955359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42045 (local address 127.0.98.1:55385)
0928 04:48:26.955511 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.955521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.955589 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.955819 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.955820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.955822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:26.955911 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.955913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.956668 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.956669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.957585 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.957588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.958111 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.958112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.958194 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.958199 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.958241 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.958272 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":51,"thread_start_us":79,"threads_started":1}
I20260928 04:48:26.958395 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.955432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55385 (local address 127.0.98.3:42045)
0928 04:48:26.955490 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.955492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.955678 (+ 186us) server_negotiation.cc:407] Connection header received
0928 04:48:26.955720 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.955721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.955736 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.955799 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:26.955989 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.956588 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.957668 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.958034 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.958090 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.958099 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.958261 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.958301 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:48:26.958509 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.955361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43289 (local address 127.0.98.1:59467)
0928 04:48:26.955705 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:26.955715 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:26.955788 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:26.955922 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:26.955922 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:26.955924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:26.956010 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.956012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.956772 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.956773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.957601 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:26.957603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.958240 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.958242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:26.958315 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.958319 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:26.958378 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:48:26.958421 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260928 04:48:26.958523 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:26.955447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59467 (local address 127.0.98.2:43289)
0928 04:48:26.955497 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:26.955500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:26.955805 (+ 305us) server_negotiation.cc:407] Connection header received
0928 04:48:26.955835 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:26.955836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:26.955854 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:26.955912 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:26.956086 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.956526 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.957712 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:26.958061 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:26.958149 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:26.958159 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:26.958398 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 04:48:26.958441 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:48:26.958680 2548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64fdab4852946e2b058c4c04ba291d7" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" is_pre_election: true
I20260928 04:48:26.958828 2548 raft_consensus.cc:2454] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 0.
I20260928 04:48:26.958817 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64fdab4852946e2b058c4c04ba291d7" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3fdf376664610b20dca7afec7e249" is_pre_election: true
I20260928 04:48:26.958997 2408 raft_consensus.cc:2454] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 0.
I20260928 04:48:26.959057 2200 leader_election.cc:302] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62bd621b36ce4624b16e107889eea5f8, 79bfbd01e2ac4ceda2b9b220df7c3f33; no voters:
I20260928 04:48:26.959240 2621 raft_consensus.cc:2781] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:26.959318 2621 raft_consensus.cc:482] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:26.959360 2621 raft_consensus.cc:3037] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Advancing to term 1
W20260928 04:48:27.005431 2455 tablet.cc:2369] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:48:27.007402 2621 raft_consensus.cc:504] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 04:48:27.007643 2621 leader_election.cc:288] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 election: Requested vote from peers 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045), 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289)
I20260928 04:48:27.007925 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64fdab4852946e2b058c4c04ba291d7" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3fdf376664610b20dca7afec7e249"
I20260928 04:48:27.007923 2548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64fdab4852946e2b058c4c04ba291d7" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33"
I20260928 04:48:27.008030 2408 raft_consensus.cc:3037] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:27.008034 2548 raft_consensus.cc:3037] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:27.008826 2548 raft_consensus.cc:2454] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 1.
I20260928 04:48:27.009085 2408 raft_consensus.cc:2454] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 1.
I20260928 04:48:27.009096 2200 leader_election.cc:302] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62bd621b36ce4624b16e107889eea5f8, 79bfbd01e2ac4ceda2b9b220df7c3f33; no voters:
I20260928 04:48:27.009235 2621 raft_consensus.cc:2781] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:27.009357 2621 raft_consensus.cc:686] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 1 LEADER]: Becoming Leader. State: Replica: 62bd621b36ce4624b16e107889eea5f8, State: Running, Role: LEADER
I20260928 04:48:27.009495 2621 consensus_queue.cc:237] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 04:48:27.010424 2110 catalog_manager.cc:5495] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "62bd621b36ce4624b16e107889eea5f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } health_report { overall_health: UNKNOWN } } }
Sep 28 04:48:27 dist-test-slave-jccs krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:27.060739 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.053899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39517 (local address 127.0.0.1:58572)
0928 04:48:27.054139 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:27.054150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:27.054219 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:27.054341 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:27.054342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:27.054637 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:27.054683 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.054685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.055340 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.055340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.055883 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.055885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.056195 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.056196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.056237 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.056797 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:27.056801 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:27.058178 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:27.059745 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:27.059751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:27.059753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:27.059937 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:27.060108 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:27.060109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:27.060109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:27.060163 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:27.060440 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:27.060447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:27.060498 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 04:48:27.060554 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:48:27.060640 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":63,"threads_started":1}
I20260928 04:48:27.060789 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.053976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58572 (local address 127.0.98.62:39517)
0928 04:48:27.054075 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:27.054077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:27.054217 (+ 140us) server_negotiation.cc:407] Connection header received
0928 04:48:27.054262 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:27.054263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:27.054272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:27.054322 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:27.054798 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.055271 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.055962 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.056133 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.056182 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.058263 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:27.058264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:27.058264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:27.058269 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:27.059676 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:27.060002 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:27.060003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:27.060003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:27.060057 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:27.060241 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:27.060242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:27.060242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:27.060346 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:27.060405 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:27.060585 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 04:48:27.060684 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":67}
I20260928 04:48:27.062819 2110 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58572:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 04:48:27.063021 2110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:27.064488 2528 tablet_service.cc:1467] Processing CreateTablet for tablet c512f160dcc9442f805cf3e39c5d636b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22a2ab2419a74e3cbff6b3f62f9287e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:48:27.064495 2248 tablet_service.cc:1467] Processing CreateTablet for tablet c512f160dcc9442f805cf3e39c5d636b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22a2ab2419a74e3cbff6b3f62f9287e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:48:27.064493 2388 tablet_service.cc:1467] Processing CreateTablet for tablet c512f160dcc9442f805cf3e39c5d636b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22a2ab2419a74e3cbff6b3f62f9287e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:48:27.064657 2528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c512f160dcc9442f805cf3e39c5d636b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:27.064687 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c512f160dcc9442f805cf3e39c5d636b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:27.064697 2248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c512f160dcc9442f805cf3e39c5d636b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:27.065904 2617 tablet_bootstrap.cc:492] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: Bootstrap starting.
I20260928 04:48:27.066228 2618 tablet_bootstrap.cc:492] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: Bootstrap starting.
I20260928 04:48:27.066229 2623 tablet_bootstrap.cc:492] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: Bootstrap starting.
I20260928 04:48:27.066493 2617 tablet_bootstrap.cc:654] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:27.066615 2618 tablet_bootstrap.cc:654] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:27.066612 2623 tablet_bootstrap.cc:654] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:27.067232 2617 tablet_bootstrap.cc:492] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: No bootstrap required, opened a new log
I20260928 04:48:27.067284 2617 ts_tablet_manager.cc:1397] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:27.067453 2617 raft_consensus.cc:348] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.067521 2618 tablet_bootstrap.cc:492] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: No bootstrap required, opened a new log
I20260928 04:48:27.067531 2617 raft_consensus.cc:374] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:27.067555 2617 raft_consensus.cc:729] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79bfbd01e2ac4ceda2b9b220df7c3f33, State: Initialized, Role: FOLLOWER
I20260928 04:48:27.067564 2618 ts_tablet_manager.cc:1397] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:27.067615 2617 consensus_queue.cc:260] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.067699 2623 tablet_bootstrap.cc:492] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: No bootstrap required, opened a new log
I20260928 04:48:27.067710 2617 ts_tablet_manager.cc:1428] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:27.067703 2618 raft_consensus.cc:348] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.067746 2623 ts_tablet_manager.cc:1397] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:27.067771 2618 raft_consensus.cc:374] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:27.067798 2618 raft_consensus.cc:729] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62bd621b36ce4624b16e107889eea5f8, State: Initialized, Role: FOLLOWER
I20260928 04:48:27.067778 2622 tablet_replica.cc:439] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } }
I20260928 04:48:27.067847 2618 consensus_queue.cc:260] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.067870 2622 tablet_replica.cc:442] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:27.067935 2618 ts_tablet_manager.cc:1428] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:27.067912 2623 raft_consensus.cc:348] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.067970 2623 raft_consensus.cc:374] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:27.067989 2623 raft_consensus.cc:729] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e3fdf376664610b20dca7afec7e249, State: Initialized, Role: FOLLOWER
I20260928 04:48:27.068033 2623 consensus_queue.cc:260] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.068039 2621 tablet_replica.cc:439] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } }
I20260928 04:48:27.068099 2621 tablet_replica.cc:442] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:27.068153 2623 ts_tablet_manager.cc:1428] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:27.068202 2625 tablet_replica.cc:439] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } }
I20260928 04:48:27.068281 2625 tablet_replica.cc:442] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:27.299305 2622 raft_consensus.cc:482] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:27.299402 2622 raft_consensus.cc:504] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.299772 2622 leader_election.cc:288] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1:37437), 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289)
I20260928 04:48:27.302651 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.300042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45443 (local address 127.0.98.2:43289)
0928 04:48:27.300087 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:27.300089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:27.300167 (+ 78us) server_negotiation.cc:407] Connection header received
0928 04:48:27.300221 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:27.300222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:27.300232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:27.300290 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:27.300464 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.300966 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.301975 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.302309 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.302361 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.302369 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:27.302532 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 04:48:27.302568 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:48:27.302659 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.299951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43289 (local address 127.0.98.3:45443)
0928 04:48:27.300088 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:27.300099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:27.300166 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:27.300309 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:27.300310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:27.300314 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:27.300381 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.300383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.301061 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.301062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.301884 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.301886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.302388 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.302389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.302459 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.302464 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:48:27.302520 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:48:27.302560 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260928 04:48:27.302830 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.299951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37437 (local address 127.0.98.3:54205)
0928 04:48:27.300073 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:27.300086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:27.300187 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:27.300378 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:27.300379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:27.300380 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:27.300466 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.300467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.301230 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.301231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.301895 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.301897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.302491 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.302492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.302573 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.302577 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:27.302632 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:48:27.302670 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260928 04:48:27.302841 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.300120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54205 (local address 127.0.98.1:37437)
0928 04:48:27.300213 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:27.300216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:27.300225 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:27.300250 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:27.300251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:27.300286 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:27.300367 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:27.300565 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.301112 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.302015 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.302412 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.302483 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.302496 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:27.302667 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 04:48:27.302733 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":45,"threads_started":1}
I20260928 04:48:27.303110 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c512f160dcc9442f805cf3e39c5d636b" candidate_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62bd621b36ce4624b16e107889eea5f8" is_pre_election: true
I20260928 04:48:27.303157 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c512f160dcc9442f805cf3e39c5d636b" candidate_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3fdf376664610b20dca7afec7e249" is_pre_election: true
I20260928 04:48:27.303249 2268 raft_consensus.cc:2454] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79bfbd01e2ac4ceda2b9b220df7c3f33 in term 0.
I20260928 04:48:27.303263 2408 raft_consensus.cc:2454] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79bfbd01e2ac4ceda2b9b220df7c3f33 in term 0.
I20260928 04:48:27.303449 2479 leader_election.cc:302] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e3fdf376664610b20dca7afec7e249, 79bfbd01e2ac4ceda2b9b220df7c3f33; no voters:
I20260928 04:48:27.303587 2622 raft_consensus.cc:2781] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:27.303629 2622 raft_consensus.cc:482] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:27.303658 2622 raft_consensus.cc:3037] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:27.304311 2622 raft_consensus.cc:504] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.304450 2622 leader_election.cc:288] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [CANDIDATE]: Term 1 election: Requested vote from peers 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1:37437), 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289)
I20260928 04:48:27.304598 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c512f160dcc9442f805cf3e39c5d636b" candidate_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62bd621b36ce4624b16e107889eea5f8"
I20260928 04:48:27.304586 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c512f160dcc9442f805cf3e39c5d636b" candidate_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3fdf376664610b20dca7afec7e249"
I20260928 04:48:27.304662 2268 raft_consensus.cc:3037] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:27.304662 2408 raft_consensus.cc:3037] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:27.305437 2408 raft_consensus.cc:2454] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79bfbd01e2ac4ceda2b9b220df7c3f33 in term 1.
I20260928 04:48:27.305541 2268 raft_consensus.cc:2454] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79bfbd01e2ac4ceda2b9b220df7c3f33 in term 1.
I20260928 04:48:27.305650 2479 leader_election.cc:302] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e3fdf376664610b20dca7afec7e249, 79bfbd01e2ac4ceda2b9b220df7c3f33; no voters:
I20260928 04:48:27.305730 2622 raft_consensus.cc:2781] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:27.305868 2622 raft_consensus.cc:686] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 LEADER]: Becoming Leader. State: Replica: 79bfbd01e2ac4ceda2b9b220df7c3f33, State: Running, Role: LEADER
I20260928 04:48:27.305958 2622 consensus_queue.cc:237] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.306406 2642 tablet_replica.cc:439] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: TxnStatusTablet state changed. Reason: New leader 79bfbd01e2ac4ceda2b9b220df7c3f33. Latest consensus state: current_term: 1 leader_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } }
I20260928 04:48:27.306524 2642 tablet_replica.cc:442] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:48:27.306747 2644 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 04:48:27.306726 2110 catalog_manager.cc:5495] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } health_report { overall_health: HEALTHY } } }
I20260928 04:48:27.306856 2644 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 04:48:27.353135 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.350567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.0.98.3:42045)
0928 04:48:27.350633 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:27.350635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:27.350649 (+ 14us) server_negotiation.cc:407] Connection header received
0928 04:48:27.350710 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:27.350710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:27.350727 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:27.350783 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:27.350906 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.351356 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.352454 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.352617 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.352665 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.352852 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:27.352948 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:27.353003 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:27.353063 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 04:48:27.353078 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:48:27.353215 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.350498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42045 (local address 127.0.0.1:43378)
0928 04:48:27.350579 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:27.350590 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:27.350638 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:27.350787 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:27.350788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:27.350789 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:48:27.350841 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.350844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.351439 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.351440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.352373 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.352376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.352683 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.352683 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.352752 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.352768 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:48:27.353006 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:48:27.353015 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:48:27.353063 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 04:48:27.353108 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 04:48:27.354995 2268 raft_consensus.cc:1261] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Refusing update from remote peer 79bfbd01e2ac4ceda2b9b220df7c3f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:27.354995 2408 raft_consensus.cc:1261] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Refusing update from remote peer 79bfbd01e2ac4ceda2b9b220df7c3f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:27.355283 2622 consensus_queue.cc:1035] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:27.355408 2642 consensus_queue.cc:1035] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:27.356340 2621 tablet_replica.cc:439] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: TxnStatusTablet state changed. Reason: New leader 79bfbd01e2ac4ceda2b9b220df7c3f33. Latest consensus state: current_term: 1 leader_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } }
I20260928 04:48:27.356447 2621 tablet_replica.cc:442] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:27.357220 2642 tablet_replica.cc:439] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } }
I20260928 04:48:27.357300 2642 tablet_replica.cc:442] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:48:27.357400 2621 tablet_replica.cc:439] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } }
I20260928 04:48:27.357458 2621 tablet_replica.cc:442] T c512f160dcc9442f805cf3e39c5d636b P 62bd621b36ce4624b16e107889eea5f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:27.357745 2649 mvcc.cc:205] Tried to move back new op lower bound from 7334178436523286528 to 7334178436325683200. Current Snapshot: MvccSnapshot[applied={T|T < 7334178436523286528}]
I20260928 04:48:27.357972 2647 mvcc.cc:205] Tried to move back new op lower bound from 7334178436523286528 to 7334178436325683200. Current Snapshot: MvccSnapshot[applied={T|T < 7334178436523286528}]
I20260928 04:48:27.358562 2625 tablet_replica.cc:439] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: TxnStatusTablet state changed. Reason: New leader 79bfbd01e2ac4ceda2b9b220df7c3f33. Latest consensus state: current_term: 1 leader_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } }
I20260928 04:48:27.358650 2625 tablet_replica.cc:442] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:27.360262 2622 tablet_replica.cc:439] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } }
I20260928 04:48:27.360345 2622 tablet_replica.cc:442] T c512f160dcc9442f805cf3e39c5d636b P 79bfbd01e2ac4ceda2b9b220df7c3f33: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:48:27.360772 2648 mvcc.cc:205] Tried to move back new op lower bound from 7334178436523286528 to 7334178436325683200. Current Snapshot: MvccSnapshot[applied={T|T < 7334178436523286528}]
I20260928 04:48:27.360874 2625 tablet_replica.cc:439] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } } }
I20260928 04:48:27.360949 2625 tablet_replica.cc:442] T c512f160dcc9442f805cf3e39c5d636b P 14e3fdf376664610b20dca7afec7e249: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:27.365217 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.362622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.0.98.1:37437)
0928 04:48:27.362663 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:27.362665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:27.362807 (+ 142us) server_negotiation.cc:407] Connection header received
0928 04:48:27.362844 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:27.362845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:27.362862 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:27.362944 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:27.363022 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.363738 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.364469 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.364754 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.364829 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.364932 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:27.365034 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:27.365102 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:27.365118 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 04:48:27.365138 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:48:27.370163 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.367535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39933 (local address 127.0.98.3:42045)
0928 04:48:27.367582 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:27.367583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:27.367700 (+ 117us) server_negotiation.cc:407] Connection header received
0928 04:48:27.367769 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:27.367770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:27.367779 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:27.367822 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:27.367946 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.368360 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.369430 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.369640 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.369693 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.369947 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:27.369971 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:27.370005 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:27.370097 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 04:48:27.370119 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:48:27.370262 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.367472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42045 (local address 127.0.98.1:39933)
0928 04:48:27.367634 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:27.367645 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:27.367705 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:27.367827 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:27.367828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:27.367829 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:48:27.367884 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.367885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.368437 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.368438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.369357 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.369360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.369722 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.369723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.369788 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.369802 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:48:27.370029 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:48:27.370031 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:48:27.370090 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:48:27.370132 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260928 04:48:27.372294 2408 raft_consensus.cc:1261] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Refusing update from remote peer 62bd621b36ce4624b16e107889eea5f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:27.372339 2548 raft_consensus.cc:1261] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Refusing update from remote peer 62bd621b36ce4624b16e107889eea5f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:27.372591 2628 consensus_queue.cc:1035] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:27.372720 2621 consensus_queue.cc:1035] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:27.380920 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.378187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56549 (local address 127.0.98.1:37437)
0928 04:48:27.378263 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:27.378265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:27.378317 (+ 52us) server_negotiation.cc:407] Connection header received
0928 04:48:27.378363 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:27.378364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:27.378372 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:27.378434 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:27.378514 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.379151 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.380197 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.380479 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.380547 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.380682 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:27.380721 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:27.380790 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:27.380830 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 04:48:27.380848 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 04:48:27.380941 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:27.378137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37437 (local address 127.0.98.3:56549)
0928 04:48:27.378263 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:27.378279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:27.378325 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:27.378435 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:27.378435 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:27.378436 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:48:27.378479 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.378479 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.379223 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.379223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.380125 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:27.380127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:27.380558 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:27.380558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:27.380601 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:27.380613 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:48:27.380794 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:48:27.380795 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:48:27.380827 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 04:48:27.380869 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260928 04:48:27.387543 2110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:27.387648 2110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:27.388396 2110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38: Sending DeleteTablet for 3 replicas of tablet d64fdab4852946e2b058c4c04ba291d7
I20260928 04:48:27.388664 2248 tablet_service.cc:1514] Processing DeleteTablet for tablet d64fdab4852946e2b058c4c04ba291d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41032
I20260928 04:48:27.388772 2528 tablet_service.cc:1514] Processing DeleteTablet for tablet d64fdab4852946e2b058c4c04ba291d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43372
I20260928 04:48:27.388988 2669 tablet_replica.cc:330] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: stopping tablet replica
I20260928 04:48:27.389081 2669 raft_consensus.cc:2229] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:27.389130 2669 raft_consensus.cc:2258] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:27.389346 2110 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:27.389497 2110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:27.389813 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet d64fdab4852946e2b058c4c04ba291d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52600
I20260928 04:48:27.390205 2670 tablet_replica.cc:330] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: stopping tablet replica
I20260928 04:48:27.390349 2670 raft_consensus.cc:2229] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:27.390455 2670 raft_consensus.cc:2258] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:27.390817 2248 tablet_service.cc:1467] Processing CreateTablet for tablet 9b45f511e58245dfbb8a294511711828 (DEFAULT_TABLE table=test-table [id=289d0ad4ad3f45dca9124acf5e7ce7f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:27.390939 2248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b45f511e58245dfbb8a294511711828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:27.390938 2670 ts_tablet_manager.cc:1905] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:27.392217 2618 tablet_bootstrap.cc:492] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8: Bootstrap starting.
I20260928 04:48:27.392621 2618 tablet_bootstrap.cc:654] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:27.392830 2670 ts_tablet_manager.cc:1918] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:27.392943 2670 log.cc:1196] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/d64fdab4852946e2b058c4c04ba291d7
I20260928 04:48:27.393226 2528 tablet_service.cc:1467] Processing CreateTablet for tablet 9b45f511e58245dfbb8a294511711828 (DEFAULT_TABLE table=test-table [id=289d0ad4ad3f45dca9124acf5e7ce7f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:27.393332 2528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b45f511e58245dfbb8a294511711828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:27.393428 2670 ts_tablet_manager.cc:1939] T d64fdab4852946e2b058c4c04ba291d7 P 14e3fdf376664610b20dca7afec7e249: Deleting consensus metadata
I20260928 04:48:27.394600 2617 tablet_bootstrap.cc:492] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Bootstrap starting.
I20260928 04:48:27.395008 2095 catalog_manager.cc:4841] TS 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289): tablet d64fdab4852946e2b058c4c04ba291d7 (table test-table [id=d90c59e22f884000aaec496e65484b6d]) successfully deleted
I20260928 04:48:27.395108 2617 tablet_bootstrap.cc:654] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:27.395136 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 9b45f511e58245dfbb8a294511711828 (DEFAULT_TABLE table=test-table [id=289d0ad4ad3f45dca9124acf5e7ce7f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:27.395220 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b45f511e58245dfbb8a294511711828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:27.395777 2617 tablet_bootstrap.cc:492] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33: No bootstrap required, opened a new log
I20260928 04:48:27.395823 2617 ts_tablet_manager.cc:1397] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:27.395951 2617 raft_consensus.cc:348] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.396008 2617 raft_consensus.cc:374] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:27.396027 2617 raft_consensus.cc:729] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79bfbd01e2ac4ceda2b9b220df7c3f33, State: Initialized, Role: FOLLOWER
I20260928 04:48:27.396067 2617 consensus_queue.cc:260] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.396153 2617 ts_tablet_manager.cc:1428] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:27.396304 2623 tablet_bootstrap.cc:492] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249: Bootstrap starting.
I20260928 04:48:27.396661 2623 tablet_bootstrap.cc:654] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:27.397404 2623 tablet_bootstrap.cc:492] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249: No bootstrap required, opened a new log
I20260928 04:48:27.397449 2623 ts_tablet_manager.cc:1397] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:27.397563 2623 raft_consensus.cc:348] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.397630 2623 raft_consensus.cc:374] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:27.397652 2623 raft_consensus.cc:729] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e3fdf376664610b20dca7afec7e249, State: Initialized, Role: FOLLOWER
I20260928 04:48:27.397689 2623 consensus_queue.cc:260] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.397771 2623 ts_tablet_manager.cc:1428] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:27.398522 2669 ts_tablet_manager.cc:1905] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:27.398674 2618 tablet_bootstrap.cc:492] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8: No bootstrap required, opened a new log
I20260928 04:48:27.398710 2618 ts_tablet_manager.cc:1397] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 04:48:27.398820 2618 raft_consensus.cc:348] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.398880 2618 raft_consensus.cc:374] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:27.398913 2618 raft_consensus.cc:729] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62bd621b36ce4624b16e107889eea5f8, State: Initialized, Role: FOLLOWER
I20260928 04:48:27.398959 2618 consensus_queue.cc:260] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.399042 2618 ts_tablet_manager.cc:1428] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:27.399540 2668 tablet_replica.cc:330] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: stopping tablet replica
I20260928 04:48:27.399642 2668 raft_consensus.cc:2229] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:27.399724 2668 raft_consensus.cc:2258] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:27.399957 2669 ts_tablet_manager.cc:1918] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:27.400015 2669 log.cc:1196] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/d64fdab4852946e2b058c4c04ba291d7
I20260928 04:48:27.400265 2669 ts_tablet_manager.cc:1939] T d64fdab4852946e2b058c4c04ba291d7 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting consensus metadata
I20260928 04:48:27.400311 2668 ts_tablet_manager.cc:1905] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:27.400822 2096 catalog_manager.cc:4841] TS 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045): tablet d64fdab4852946e2b058c4c04ba291d7 (table test-table [id=d90c59e22f884000aaec496e65484b6d]) successfully deleted
I20260928 04:48:27.401700 2668 ts_tablet_manager.cc:1918] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:27.401764 2668 log.cc:1196] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/d64fdab4852946e2b058c4c04ba291d7
I20260928 04:48:27.401988 2668 ts_tablet_manager.cc:1939] T d64fdab4852946e2b058c4c04ba291d7 P 62bd621b36ce4624b16e107889eea5f8: Deleting consensus metadata
I20260928 04:48:27.402386 2097 catalog_manager.cc:4841] TS 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1:37437): tablet d64fdab4852946e2b058c4c04ba291d7 (table test-table [id=d90c59e22f884000aaec496e65484b6d]) successfully deleted
I20260928 04:48:27.449550 2621 raft_consensus.cc:482] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:27.449666 2621 raft_consensus.cc:504] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.449885 2621 leader_election.cc:288] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289), 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045)
I20260928 04:48:27.450047 2548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b45f511e58245dfbb8a294511711828" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" is_pre_election: true
I20260928 04:48:27.450052 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b45f511e58245dfbb8a294511711828" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3fdf376664610b20dca7afec7e249" is_pre_election: true
I20260928 04:48:27.450153 2408 raft_consensus.cc:2454] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 0.
I20260928 04:48:27.450155 2548 raft_consensus.cc:2454] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 0.
I20260928 04:48:27.450295 2200 leader_election.cc:302] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62bd621b36ce4624b16e107889eea5f8, 79bfbd01e2ac4ceda2b9b220df7c3f33; no voters:
I20260928 04:48:27.450371 2621 raft_consensus.cc:2781] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:27.450413 2621 raft_consensus.cc:482] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:27.450434 2621 raft_consensus.cc:3037] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:27.451222 2621 raft_consensus.cc:504] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.451342 2621 leader_election.cc:288] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 election: Requested vote from peers 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289), 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045)
I20260928 04:48:27.451508 2548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b45f511e58245dfbb8a294511711828" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33"
I20260928 04:48:27.451500 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b45f511e58245dfbb8a294511711828" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3fdf376664610b20dca7afec7e249"
I20260928 04:48:27.451577 2548 raft_consensus.cc:3037] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:27.451596 2408 raft_consensus.cc:3037] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:27.452229 2548 raft_consensus.cc:2454] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 1.
I20260928 04:48:27.452234 2408 raft_consensus.cc:2454] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 1.
I20260928 04:48:27.452387 2200 leader_election.cc:302] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62bd621b36ce4624b16e107889eea5f8, 79bfbd01e2ac4ceda2b9b220df7c3f33; no voters:
I20260928 04:48:27.452477 2621 raft_consensus.cc:2781] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:27.452522 2621 raft_consensus.cc:686] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 1 LEADER]: Becoming Leader. State: Replica: 62bd621b36ce4624b16e107889eea5f8, State: Running, Role: LEADER
I20260928 04:48:27.452569 2621 consensus_queue.cc:237] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:27.453132 2111 catalog_manager.cc:5495] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "62bd621b36ce4624b16e107889eea5f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } health_report { overall_health: UNKNOWN } } }
I20260928 04:48:27.458763 2408 raft_consensus.cc:1261] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Refusing update from remote peer 62bd621b36ce4624b16e107889eea5f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:27.458782 2548 raft_consensus.cc:1261] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Refusing update from remote peer 62bd621b36ce4624b16e107889eea5f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:27.458936 2621 consensus_queue.cc:1035] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:27.459023 2663 consensus_queue.cc:1035] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 04:48:27.463486 2649 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 04:48:28.981272 2110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:28.981367 2110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:28.982036 2110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38: Sending DeleteTablet for 3 replicas of tablet 9b45f511e58245dfbb8a294511711828
I20260928 04:48:28.982285 2248 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b45f511e58245dfbb8a294511711828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41032
I20260928 04:48:28.982296 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b45f511e58245dfbb8a294511711828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52600
I20260928 04:48:28.982288 2528 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b45f511e58245dfbb8a294511711828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43372
I20260928 04:48:28.982470 2700 tablet_replica.cc:330] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8: stopping tablet replica
I20260928 04:48:28.982527 2700 raft_consensus.cc:2229] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:28.982681 2700 raft_consensus.cc:2258] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:28.982933 2700 ts_tablet_manager.cc:1905] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:28.984429 2700 ts_tablet_manager.cc:1918] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:48:28.984493 2700 log.cc:1196] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/9b45f511e58245dfbb8a294511711828
I20260928 04:48:28.984750 2700 ts_tablet_manager.cc:1939] T 9b45f511e58245dfbb8a294511711828 P 62bd621b36ce4624b16e107889eea5f8: Deleting consensus metadata
I20260928 04:48:28.985162 2097 catalog_manager.cc:4841] TS 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1:37437): tablet 9b45f511e58245dfbb8a294511711828 (table test-table [id=289d0ad4ad3f45dca9124acf5e7ce7f8]) successfully deleted
I20260928 04:48:28.986757 2701 tablet_replica.cc:330] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249: stopping tablet replica
I20260928 04:48:28.986807 2701 raft_consensus.cc:2229] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:28.986855 2701 raft_consensus.cc:2258] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:28.987104 2701 ts_tablet_manager.cc:1905] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:28.988600 2701 ts_tablet_manager.cc:1918] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:48:28.988662 2701 log.cc:1196] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/9b45f511e58245dfbb8a294511711828
I20260928 04:48:28.988860 2701 ts_tablet_manager.cc:1939] T 9b45f511e58245dfbb8a294511711828 P 14e3fdf376664610b20dca7afec7e249: Deleting consensus metadata
I20260928 04:48:28.989249 2095 catalog_manager.cc:4841] TS 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289): tablet 9b45f511e58245dfbb8a294511711828 (table test-table [id=289d0ad4ad3f45dca9124acf5e7ce7f8]) successfully deleted
I20260928 04:48:28.989266 2702 tablet_replica.cc:330] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33: stopping tablet replica
I20260928 04:48:28.989323 2702 raft_consensus.cc:2229] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:28.989358 2702 raft_consensus.cc:2258] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:28.989560 2702 ts_tablet_manager.cc:1905] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:28.990808 2702 ts_tablet_manager.cc:1918] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:48:28.990857 2702 log.cc:1196] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/9b45f511e58245dfbb8a294511711828
I20260928 04:48:28.991056 2702 ts_tablet_manager.cc:1939] T 9b45f511e58245dfbb8a294511711828 P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting consensus metadata
I20260928 04:48:28.991384 2096 catalog_manager.cc:4841] TS 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045): tablet 9b45f511e58245dfbb8a294511711828 (table test-table [id=289d0ad4ad3f45dca9124acf5e7ce7f8]) successfully deleted
Sep 28 04:48:28 dist-test-slave-jccs krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:48:29 dist-test-slave-jccs krb5kdc[2066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:29.004908 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:28.998099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.0.98.62:39517)
0928 04:48:28.998251 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:28.998253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:28.998262 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:28.998337 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:28.998338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:28.998349 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:28.998418 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:28.998975 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:28.999474 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:29.000203 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:29.000416 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:29.000497 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:29.002611 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:29.002613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:29.002613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:29.002620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:29.003856 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:29.004202 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:29.004203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:29.004203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:29.004258 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:29.004429 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:29.004430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:29.004431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:29.004532 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:29.004633 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:29.004713 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 04:48:29.004789 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 04:48:29.006284 2110 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:29.006417 2110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:29.007567 2248 tablet_service.cc:1467] Processing CreateTablet for tablet 371c95e454ec48ae8319b7d1f89b0bed (DEFAULT_TABLE table=test-table [id=65986a8b13cc4d179cd8a486ab835352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:29.007616 2388 tablet_service.cc:1467] Processing CreateTablet for tablet 371c95e454ec48ae8319b7d1f89b0bed (DEFAULT_TABLE table=test-table [id=65986a8b13cc4d179cd8a486ab835352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:29.007659 2528 tablet_service.cc:1467] Processing CreateTablet for tablet 371c95e454ec48ae8319b7d1f89b0bed (DEFAULT_TABLE table=test-table [id=65986a8b13cc4d179cd8a486ab835352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:29.007694 2248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371c95e454ec48ae8319b7d1f89b0bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:29.007742 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371c95e454ec48ae8319b7d1f89b0bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:29.007761 2528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371c95e454ec48ae8319b7d1f89b0bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:29.008636 2711 tablet_bootstrap.cc:492] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249: Bootstrap starting.
I20260928 04:48:29.009013 2711 tablet_bootstrap.cc:654] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:29.009117 2712 tablet_bootstrap.cc:492] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33: Bootstrap starting.
I20260928 04:48:29.009513 2712 tablet_bootstrap.cc:654] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:29.009624 2711 tablet_bootstrap.cc:492] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249: No bootstrap required, opened a new log
I20260928 04:48:29.009677 2711 ts_tablet_manager.cc:1397] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:29.009814 2713 tablet_bootstrap.cc:492] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8: Bootstrap starting.
I20260928 04:48:29.009821 2711 raft_consensus.cc:348] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.009891 2711 raft_consensus.cc:374] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:29.009919 2711 raft_consensus.cc:729] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e3fdf376664610b20dca7afec7e249, State: Initialized, Role: FOLLOWER
I20260928 04:48:29.009992 2711 consensus_queue.cc:260] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.010157 2713 tablet_bootstrap.cc:654] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:29.010159 2711 ts_tablet_manager.cc:1428] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:29.010551 2712 tablet_bootstrap.cc:492] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33: No bootstrap required, opened a new log
I20260928 04:48:29.010605 2712 ts_tablet_manager.cc:1397] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:29.011018 2713 tablet_bootstrap.cc:492] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8: No bootstrap required, opened a new log
I20260928 04:48:29.011080 2713 ts_tablet_manager.cc:1397] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:29.011178 2713 raft_consensus.cc:348] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.011227 2713 raft_consensus.cc:374] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:29.011240 2713 raft_consensus.cc:729] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62bd621b36ce4624b16e107889eea5f8, State: Initialized, Role: FOLLOWER
I20260928 04:48:29.011292 2713 consensus_queue.cc:260] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.011384 2713 ts_tablet_manager.cc:1428] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:29.011474 2712 raft_consensus.cc:348] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.011546 2712 raft_consensus.cc:374] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:29.011569 2712 raft_consensus.cc:729] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79bfbd01e2ac4ceda2b9b220df7c3f33, State: Initialized, Role: FOLLOWER
I20260928 04:48:29.011621 2712 consensus_queue.cc:260] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.011715 2712 ts_tablet_manager.cc:1428] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 04:48:29.089025 2681 raft_consensus.cc:482] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:29.089126 2681 raft_consensus.cc:504] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.089356 2681 leader_election.cc:288] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1:37437), 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289)
I20260928 04:48:29.089547 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371c95e454ec48ae8319b7d1f89b0bed" candidate_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3fdf376664610b20dca7afec7e249" is_pre_election: true
I20260928 04:48:29.089556 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371c95e454ec48ae8319b7d1f89b0bed" candidate_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62bd621b36ce4624b16e107889eea5f8" is_pre_election: true
I20260928 04:48:29.089643 2268 raft_consensus.cc:2454] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79bfbd01e2ac4ceda2b9b220df7c3f33 in term 0.
I20260928 04:48:29.089643 2408 raft_consensus.cc:2454] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79bfbd01e2ac4ceda2b9b220df7c3f33 in term 0.
I20260928 04:48:29.089807 2479 leader_election.cc:302] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e3fdf376664610b20dca7afec7e249, 79bfbd01e2ac4ceda2b9b220df7c3f33; no voters:
I20260928 04:48:29.089916 2681 raft_consensus.cc:2781] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:29.089956 2681 raft_consensus.cc:482] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:29.090003 2681 raft_consensus.cc:3037] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:29.090678 2681 raft_consensus.cc:504] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.090809 2681 leader_election.cc:288] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [CANDIDATE]: Term 1 election: Requested vote from peers 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1:37437), 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289)
I20260928 04:48:29.090925 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371c95e454ec48ae8319b7d1f89b0bed" candidate_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62bd621b36ce4624b16e107889eea5f8"
I20260928 04:48:29.090942 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371c95e454ec48ae8319b7d1f89b0bed" candidate_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3fdf376664610b20dca7afec7e249"
I20260928 04:48:29.091007 2408 raft_consensus.cc:3037] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:29.091007 2268 raft_consensus.cc:3037] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:29.091713 2268 raft_consensus.cc:2454] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79bfbd01e2ac4ceda2b9b220df7c3f33 in term 1.
I20260928 04:48:29.091713 2408 raft_consensus.cc:2454] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79bfbd01e2ac4ceda2b9b220df7c3f33 in term 1.
I20260928 04:48:29.091881 2481 leader_election.cc:302] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62bd621b36ce4624b16e107889eea5f8, 79bfbd01e2ac4ceda2b9b220df7c3f33; no voters:
I20260928 04:48:29.091985 2681 raft_consensus.cc:2781] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:29.092044 2681 raft_consensus.cc:686] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 LEADER]: Becoming Leader. State: Replica: 79bfbd01e2ac4ceda2b9b220df7c3f33, State: Running, Role: LEADER
I20260928 04:48:29.092075 2681 consensus_queue.cc:237] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.092674 2110 catalog_manager.cc:5495] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } health_report { overall_health: HEALTHY } } }
I20260928 04:48:29.104238 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:29.102145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.0.98.3:42045)
0928 04:48:29.102277 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:29.102279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:29.102285 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:48:29.102302 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:29.102303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:29.102311 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:29.102364 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:29.102476 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:29.102930 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:29.103601 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:29.103780 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:29.103826 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:29.103942 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:29.103964 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:29.104019 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:29.104076 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 04:48:29.104102 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260928 04:48:29.108345 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:29.106142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50497 (local address 127.0.98.3:42045)
0928 04:48:29.106188 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:29.106190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:29.106303 (+ 113us) server_negotiation.cc:407] Connection header received
0928 04:48:29.106341 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:29.106342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:29.106349 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:29.106403 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:29.106531 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:29.107020 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:29.107808 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:29.108015 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:29.108065 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:29.108167 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:29.108189 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:29.108226 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:29.108270 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 04:48:29.108285 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:48:29.108393 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:29.106116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42045 (local address 127.0.98.3:50497)
0928 04:48:29.106241 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:29.106251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:29.106306 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:29.106406 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:29.106407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:29.106409 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:48:29.106465 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:29.106467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:29.107076 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:29.107077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:29.107745 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:29.107747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:29.108074 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:29.108075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:29.108110 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:29.108114 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:48:29.108235 (+ 121us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:48:29.108237 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:48:29.108270 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 04:48:29.108299 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":88,"threads_started":1}
I20260928 04:48:29.110153 2408 raft_consensus.cc:1261] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Refusing update from remote peer 79bfbd01e2ac4ceda2b9b220df7c3f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:29.110183 2268 raft_consensus.cc:1261] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Refusing update from remote peer 79bfbd01e2ac4ceda2b9b220df7c3f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:29.110431 2681 consensus_queue.cc:1035] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:29.110524 2678 consensus_queue.cc:1035] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:29.120198 2110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:29.120294 2110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:29.120839 2110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38: Sending DeleteTablet for 3 replicas of tablet 371c95e454ec48ae8319b7d1f89b0bed
I20260928 04:48:29.121094 2248 tablet_service.cc:1514] Processing DeleteTablet for tablet 371c95e454ec48ae8319b7d1f89b0bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41032
I20260928 04:48:29.121135 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet 371c95e454ec48ae8319b7d1f89b0bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52600
I20260928 04:48:29.121133 2528 tablet_service.cc:1514] Processing DeleteTablet for tablet 371c95e454ec48ae8319b7d1f89b0bed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43372
I20260928 04:48:29.121214 2700 tablet_replica.cc:330] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8: stopping tablet replica
I20260928 04:48:29.121243 2701 tablet_replica.cc:330] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249: stopping tablet replica
I20260928 04:48:29.121243 2702 tablet_replica.cc:330] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33: stopping tablet replica
I20260928 04:48:29.121258 2700 raft_consensus.cc:2229] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:29.121289 2700 raft_consensus.cc:2258] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:29.121284 2701 raft_consensus.cc:2229] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:29.121289 2702 raft_consensus.cc:2229] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:29.121332 2701 raft_consensus.cc:2258] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:29.121366 2702 raft_consensus.cc:2258] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:29.121594 2700 ts_tablet_manager.cc:1905] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:29.121618 2702 ts_tablet_manager.cc:1905] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:29.121774 2701 ts_tablet_manager.cc:1905] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:29.123144 2702 ts_tablet_manager.cc:1918] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:29.123150 2700 ts_tablet_manager.cc:1918] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:29.123199 2702 log.cc:1196] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/371c95e454ec48ae8319b7d1f89b0bed
I20260928 04:48:29.123201 2700 log.cc:1196] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/371c95e454ec48ae8319b7d1f89b0bed
I20260928 04:48:29.123425 2700 ts_tablet_manager.cc:1939] T 371c95e454ec48ae8319b7d1f89b0bed P 62bd621b36ce4624b16e107889eea5f8: Deleting consensus metadata
I20260928 04:48:29.123426 2702 ts_tablet_manager.cc:1939] T 371c95e454ec48ae8319b7d1f89b0bed P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting consensus metadata
I20260928 04:48:29.123528 2110 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:29.123689 2110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:29.123878 2701 ts_tablet_manager.cc:1918] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:29.123934 2701 log.cc:1196] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/371c95e454ec48ae8319b7d1f89b0bed
I20260928 04:48:29.124140 2701 ts_tablet_manager.cc:1939] T 371c95e454ec48ae8319b7d1f89b0bed P 14e3fdf376664610b20dca7afec7e249: Deleting consensus metadata
I20260928 04:48:29.124193 2096 catalog_manager.cc:4841] TS 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045): tablet 371c95e454ec48ae8319b7d1f89b0bed (table test-table [id=65986a8b13cc4d179cd8a486ab835352]) successfully deleted
I20260928 04:48:29.124449 2095 catalog_manager.cc:4841] TS 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289): tablet 371c95e454ec48ae8319b7d1f89b0bed (table test-table [id=65986a8b13cc4d179cd8a486ab835352]) successfully deleted
I20260928 04:48:29.124832 2097 catalog_manager.cc:4841] TS 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1:37437): tablet 371c95e454ec48ae8319b7d1f89b0bed (table test-table [id=65986a8b13cc4d179cd8a486ab835352]) successfully deleted
I20260928 04:48:29.125027 2248 tablet_service.cc:1467] Processing CreateTablet for tablet dc103af56082444a9da2b978029ed57c (DEFAULT_TABLE table=test-table [id=3b8a80ccc00346ac941d3ce2699d8b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:29.125087 2528 tablet_service.cc:1467] Processing CreateTablet for tablet dc103af56082444a9da2b978029ed57c (DEFAULT_TABLE table=test-table [id=3b8a80ccc00346ac941d3ce2699d8b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:29.125133 2248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc103af56082444a9da2b978029ed57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:29.125211 2528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc103af56082444a9da2b978029ed57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:29.126281 2712 tablet_bootstrap.cc:492] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33: Bootstrap starting.
I20260928 04:48:29.126281 2713 tablet_bootstrap.cc:492] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8: Bootstrap starting.
I20260928 04:48:29.126560 2713 tablet_bootstrap.cc:654] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:29.126611 2712 tablet_bootstrap.cc:654] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:29.126849 2388 tablet_service.cc:1467] Processing CreateTablet for tablet dc103af56082444a9da2b978029ed57c (DEFAULT_TABLE table=test-table [id=3b8a80ccc00346ac941d3ce2699d8b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:29.126935 2388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc103af56082444a9da2b978029ed57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:29.127070 2713 tablet_bootstrap.cc:492] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8: No bootstrap required, opened a new log
I20260928 04:48:29.127120 2713 ts_tablet_manager.cc:1397] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 04:48:29.127214 2713 raft_consensus.cc:348] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.127274 2713 raft_consensus.cc:374] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:29.127292 2713 raft_consensus.cc:729] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62bd621b36ce4624b16e107889eea5f8, State: Initialized, Role: FOLLOWER
I20260928 04:48:29.127336 2713 consensus_queue.cc:260] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.127415 2713 ts_tablet_manager.cc:1428] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:29.128012 2711 tablet_bootstrap.cc:492] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249: Bootstrap starting.
I20260928 04:48:29.128383 2711 tablet_bootstrap.cc:654] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:29.128445 2712 tablet_bootstrap.cc:492] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33: No bootstrap required, opened a new log
I20260928 04:48:29.128484 2712 ts_tablet_manager.cc:1397] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:29.128594 2712 raft_consensus.cc:348] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.128646 2712 raft_consensus.cc:374] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:29.128684 2712 raft_consensus.cc:729] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79bfbd01e2ac4ceda2b9b220df7c3f33, State: Initialized, Role: FOLLOWER
I20260928 04:48:29.128720 2712 consensus_queue.cc:260] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.128787 2712 ts_tablet_manager.cc:1428] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:29.129053 2711 tablet_bootstrap.cc:492] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249: No bootstrap required, opened a new log
I20260928 04:48:29.129112 2711 ts_tablet_manager.cc:1397] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:29.129264 2711 raft_consensus.cc:348] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.129335 2711 raft_consensus.cc:374] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:29.129357 2711 raft_consensus.cc:729] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e3fdf376664610b20dca7afec7e249, State: Initialized, Role: FOLLOWER
I20260928 04:48:29.129400 2711 consensus_queue.cc:260] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.129485 2711 ts_tablet_manager.cc:1428] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:29.338510 2663 raft_consensus.cc:482] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:29.338608 2663 raft_consensus.cc:504] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.338819 2663 leader_election.cc:288] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289), 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045)
I20260928 04:48:29.339051 2548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc103af56082444a9da2b978029ed57c" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" is_pre_election: true
I20260928 04:48:29.339051 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc103af56082444a9da2b978029ed57c" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3fdf376664610b20dca7afec7e249" is_pre_election: true
I20260928 04:48:29.339150 2408 raft_consensus.cc:2454] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 0.
I20260928 04:48:29.339159 2548 raft_consensus.cc:2454] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 0.
I20260928 04:48:29.339339 2200 leader_election.cc:302] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62bd621b36ce4624b16e107889eea5f8, 79bfbd01e2ac4ceda2b9b220df7c3f33; no voters:
I20260928 04:48:29.339454 2663 raft_consensus.cc:2781] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:29.339519 2663 raft_consensus.cc:482] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:29.339550 2663 raft_consensus.cc:3037] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:29.340272 2663 raft_consensus.cc:504] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.340426 2663 leader_election.cc:288] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 election: Requested vote from peers 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289), 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045)
I20260928 04:48:29.340600 2548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc103af56082444a9da2b978029ed57c" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33"
I20260928 04:48:29.340579 2408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc103af56082444a9da2b978029ed57c" candidate_uuid: "62bd621b36ce4624b16e107889eea5f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3fdf376664610b20dca7afec7e249"
I20260928 04:48:29.340694 2548 raft_consensus.cc:3037] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:29.340694 2408 raft_consensus.cc:3037] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:29.341353 2408 raft_consensus.cc:2454] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 1.
I20260928 04:48:29.341434 2548 raft_consensus.cc:2454] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62bd621b36ce4624b16e107889eea5f8 in term 1.
I20260928 04:48:29.341568 2199 leader_election.cc:302] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e3fdf376664610b20dca7afec7e249, 62bd621b36ce4624b16e107889eea5f8; no voters:
I20260928 04:48:29.341657 2663 raft_consensus.cc:2781] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:29.341734 2663 raft_consensus.cc:686] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [term 1 LEADER]: Becoming Leader. State: Replica: 62bd621b36ce4624b16e107889eea5f8, State: Running, Role: LEADER
I20260928 04:48:29.341797 2663 consensus_queue.cc:237] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } }
I20260928 04:48:29.342402 2110 catalog_manager.cc:5495] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "62bd621b36ce4624b16e107889eea5f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62bd621b36ce4624b16e107889eea5f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 } health_report { overall_health: UNKNOWN } } }
I20260928 04:48:29.358816 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:29.356606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.0.98.1:37437)
0928 04:48:29.356775 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:29.356777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:29.356782 (+ 5us) server_negotiation.cc:407] Connection header received
0928 04:48:29.356798 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:29.356798 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:29.356806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:29.356874 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:29.356977 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:29.357477 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:29.358175 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:29.358373 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:29.358443 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:29.358577 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:29.358614 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:29.358669 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:29.358723 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 04:48:29.358742 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
I20260928 04:48:29.360707 2548 raft_consensus.cc:1261] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Refusing update from remote peer 62bd621b36ce4624b16e107889eea5f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:29.360755 2408 raft_consensus.cc:1261] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Refusing update from remote peer 62bd621b36ce4624b16e107889eea5f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:29.360921 2663 consensus_queue.cc:1035] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79bfbd01e2ac4ceda2b9b220df7c3f33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:29.361013 2691 consensus_queue.cc:1035] T dc103af56082444a9da2b978029ed57c P 62bd621b36ce4624b16e107889eea5f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e3fdf376664610b20dca7afec7e249" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 04:48:29.364996 2682 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 04:48:30.881584 2111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:30.881676 2111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:30.882406 2111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa272a547d524242ac0132becf5a7d38: Sending DeleteTablet for 3 replicas of tablet dc103af56082444a9da2b978029ed57c
I20260928 04:48:30.882619 2248 tablet_service.cc:1514] Processing DeleteTablet for tablet dc103af56082444a9da2b978029ed57c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41032
I20260928 04:48:30.882655 2388 tablet_service.cc:1514] Processing DeleteTablet for tablet dc103af56082444a9da2b978029ed57c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52600
I20260928 04:48:30.882790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 2183
I20260928 04:48:30.882835 2756 tablet_replica.cc:330] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249: stopping tablet replica
I20260928 04:48:30.882813 2528 tablet_service.cc:1514] Processing DeleteTablet for tablet dc103af56082444a9da2b978029ed57c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43372
I20260928 04:48:30.884377 2758 tablet_replica.cc:330] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33: stopping tablet replica
I20260928 04:48:30.884464 2758 raft_consensus.cc:2229] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:30.884514 2758 raft_consensus.cc:2258] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:30.884770 2758 ts_tablet_manager.cc:1905] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:30.882900 2756 raft_consensus.cc:2229] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:30.885228 2756 raft_consensus.cc:2258] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:30.885462 2756 ts_tablet_manager.cc:1905] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:30.886685 2758 ts_tablet_manager.cc:1918] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:48:30.886795 2758 log.cc:1196] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/dc103af56082444a9da2b978029ed57c
I20260928 04:48:30.887032 2758 ts_tablet_manager.cc:1939] T dc103af56082444a9da2b978029ed57c P 79bfbd01e2ac4ceda2b9b220df7c3f33: Deleting consensus metadata
I20260928 04:48:30.887142 2756 ts_tablet_manager.cc:1918] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 04:48:30.887200 2756 log.cc:1196] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/dc103af56082444a9da2b978029ed57c
I20260928 04:48:30.887372 2096 catalog_manager.cc:4841] TS 79bfbd01e2ac4ceda2b9b220df7c3f33 (127.0.98.3:42045): tablet dc103af56082444a9da2b978029ed57c (table test-table [id=3b8a80ccc00346ac941d3ce2699d8b07]) successfully deleted
I20260928 04:48:30.887414 2756 ts_tablet_manager.cc:1939] T dc103af56082444a9da2b978029ed57c P 14e3fdf376664610b20dca7afec7e249: Deleting consensus metadata
I20260928 04:48:30.887727 2095 catalog_manager.cc:4841] TS 14e3fdf376664610b20dca7afec7e249 (127.0.98.2:43289): tablet dc103af56082444a9da2b978029ed57c (table test-table [id=3b8a80ccc00346ac941d3ce2699d8b07]) successfully deleted
W20260928 04:48:30.889717 2097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37437 (error 108)
I20260928 04:48:30.889925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 2323
I20260928 04:48:30.890687 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:30.890074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37437 (local address 127.0.0.1:36200)
0928 04:48:30.890616 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:30.890656 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":53,"threads_started":1}
W20260928 04:48:30.890810 2097 catalog_manager.cc:4584] TS 62bd621b36ce4624b16e107889eea5f8 (127.0.98.1:37437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc103af56082444a9da2b978029ed57c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37437: connect: Connection refused (error 111)
I20260928 04:48:30.895133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 2463
I20260928 04:48:30.900249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 2079
2026-09-28T04:48:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6010 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:30 dist-test-slave-jccs krb5kdc[2763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:30 dist-test-slave-jccs krb5kdc[2763](info): set up 2 sockets
Sep 28 04:48:30 dist-test-slave-jccs krb5kdc[2763](info): commencing operation
krb5kdc: starting...
W20260928 04:48:31.896876 392 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:31 dist-test-slave-jccs krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:32.004112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:32.079012 2776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:32.079245 2776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:32.079282 2776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:32.080418 2776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:32.080490 2776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:32.080530 2776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:32.080567 2776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:32.080591 2776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:32.082049 2776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:32.082306 2776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:32.082504 2776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:32.087658 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:32.087761 2776 server_base.cc:1034] running on GCE node
W20260928 04:48:32.087726 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:32.087707 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:32.088119 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:32.088347 2776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:32.089475 2776 hybrid_clock.cc:648] HybridClock initialized: now 1790570912089457 us; error 36 us; skew 500 ppm
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:32.095683 2776 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:32.096114 2776 webserver.cc:460] Webserver started at http://127.0.98.62:40835/ using document root <none> and password file <none>
I20260928 04:48:32.096340 2776 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:32.096383 2776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:32.096544 2776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:32.097625 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "0a5cedf23cca499e8c0990aa49d886f7"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "3a46de8ae339f4235eaf7cd2969c3b91"
server_key_iv: "4753e4ecd320bd6fe72797e72ae88daf"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.097963 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a5cedf23cca499e8c0990aa49d886f7"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "3a46de8ae339f4235eaf7cd2969c3b91"
server_key_iv: "4753e4ecd320bd6fe72797e72ae88daf"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.099170 2776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:32.099835 2791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:32.100028 2776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:32.100106 2776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "0a5cedf23cca499e8c0990aa49d886f7"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "3a46de8ae339f4235eaf7cd2969c3b91"
server_key_iv: "4753e4ecd320bd6fe72797e72ae88daf"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.100155 2776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:32.119333 2776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:32.121004 2776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:32.121105 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:32.124785 2776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42583
I20260928 04:48:32.124814 2843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42583 every 8 connection(s)
I20260928 04:48:32.125291 2776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:32.125922 2844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:32.128108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 2776
I20260928 04:48:32.128201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:32.128180 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7: Bootstrap starting.
I20260928 04:48:32.128369 392 external_mini_cluster.cc:1428] Setting key 106cf4a0c913de09748556f8bcb611bb
I20260928 04:48:32.128888 2844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:32.129267 2844 log.cc:824] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:32.129979 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7: No bootstrap required, opened a new log
I20260928 04:48:32.132156 2844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5cedf23cca499e8c0990aa49d886f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42583 } }
I20260928 04:48:32.132309 2844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:32.132341 2844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a5cedf23cca499e8c0990aa49d886f7, State: Initialized, Role: FOLLOWER
I20260928 04:48:32.132462 2844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5cedf23cca499e8c0990aa49d886f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42583 } }
I20260928 04:48:32.132555 2844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:32.132606 2844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:32.132651 2844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:32.133224 2844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5cedf23cca499e8c0990aa49d886f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42583 } }
I20260928 04:48:32.133334 2844 leader_election.cc:302] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a5cedf23cca499e8c0990aa49d886f7; no voters:
I20260928 04:48:32.133487 2844 leader_election.cc:288] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:32.133656 2850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:32.133745 2844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:32.133808 2850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [term 1 LEADER]: Becoming Leader. State: Replica: 0a5cedf23cca499e8c0990aa49d886f7, State: Running, Role: LEADER
I20260928 04:48:32.133957 2850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5cedf23cca499e8c0990aa49d886f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42583 } }
I20260928 04:48:32.134344 2849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a5cedf23cca499e8c0990aa49d886f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5cedf23cca499e8c0990aa49d886f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42583 } } }
I20260928 04:48:32.134434 2849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:32.134593 2851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a5cedf23cca499e8c0990aa49d886f7. Latest consensus state: current_term: 1 leader_uuid: "0a5cedf23cca499e8c0990aa49d886f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a5cedf23cca499e8c0990aa49d886f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42583 } } }
I20260928 04:48:32.134652 2851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7 [sys.catalog]: This master's current role is: LEADER
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260928 04:48:32.135483 2865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:48:32.135541 2865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:48:32.135601 2858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:32.135763 2858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:32.137125 2858 catalog_manager.cc:1347] Generated new cluster ID: 08a4830377234169b779908ef2880379
I20260928 04:48:32.137171 2858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:32.139396 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:32.128852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.0.98.62:42583)
0928 04:48:32.129044 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:32.129053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:32.129087 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:48:32.129602 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:32.129610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:32.129663 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:32.129778 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:32.130584 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.131421 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.132507 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.132990 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.133066 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:32.135929 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:32.135933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:32.135934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:32.135944 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:32.137891 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:32.138276 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:32.138279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:32.138279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:32.138383 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:32.138597 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:32.138598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:32.138598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:32.138850 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:32.138984 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:32.139096 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 04:48:32.139211 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260928 04:48:32.146679 2858 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:32.147130 2858 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:32.149519 2858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a5cedf23cca499e8c0990aa49d886f7: Generated new TSK 0
I20260928 04:48:32.149664 2858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:48:32.173681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42583
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:32.250699 2870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:32.250881 2870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:32.250903 2870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:32.252049 2870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:32.252086 2870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:32.252110 2870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:32.253573 2870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:32.253794 2870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:32.254000 2870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:32.258626 2876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:32.258797 2870 server_base.cc:1034] running on GCE node
W20260928 04:48:32.258594 2875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:32.258594 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:32.259090 2870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:32.259356 2870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:32.260506 2870 hybrid_clock.cc:648] HybridClock initialized: now 1790570912260510 us; error 41 us; skew 500 ppm
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:32.262917 2870 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:48:32.263317 2870 webserver.cc:460] Webserver started at http://127.0.98.1:45299/ using document root <none> and password file <none>
I20260928 04:48:32.263545 2870 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:32.263610 2870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:32.263754 2870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:32.264680 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "134fc76632d6411e97d99940089517ff"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "afbe406db368de30fa53067ef46ec5b4"
server_key_iv: "d80b9404ce774bf24dabd1019f1c4dc4"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.265046 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "134fc76632d6411e97d99940089517ff"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "afbe406db368de30fa53067ef46ec5b4"
server_key_iv: "d80b9404ce774bf24dabd1019f1c4dc4"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.266286 2870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:32.266902 2885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:32.267059 2870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:32.267150 2870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "134fc76632d6411e97d99940089517ff"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "afbe406db368de30fa53067ef46ec5b4"
server_key_iv: "d80b9404ce774bf24dabd1019f1c4dc4"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.267237 2870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:32.276494 2870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:32.278295 2870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:32.278437 2870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:32.278637 2870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:32.278904 2870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:32.278954 2870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:32.278993 2870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:32.279018 2870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:32.284476 2870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33651
I20260928 04:48:32.284554 2998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33651 every 8 connection(s)
I20260928 04:48:32.284843 2870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:48:32.287585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 2870
I20260928 04:48:32.287659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:32.287834 392 external_mini_cluster.cc:1428] Setting key 85946a479942f41ad0792c54de44ef9e
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:32.293881 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:32.285498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56501 (local address 127.0.98.62:42583)
0928 04:48:32.285564 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:32.285566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:32.285749 (+ 183us) server_negotiation.cc:407] Connection header received
0928 04:48:32.286604 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:32.286605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:32.286615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:32.286672 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:32.287192 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.287734 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.288854 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.289055 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.289130 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:32.291293 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:32.291295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:32.291295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:32.291302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:32.292625 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:32.292986 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:32.292987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:32.292987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:32.293039 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:32.293317 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:32.293318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:32.293318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:32.293424 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:32.293491 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:32.293728 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 04:48:32.293803 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:48:32.294054 3001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:32.285504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42583 (local address 127.0.98.1:56501)
0928 04:48:32.285643 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:32.285673 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:32.286513 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:32.286721 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:32.286724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:32.286951 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:32.287127 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:32.287131 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.287833 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.287834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:32.288764 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:32.288766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.289141 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.289142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:32.289285 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:32.289759 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:32.289768 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:32.291199 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:32.292716 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:32.292721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:32.292723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:32.292918 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:32.293106 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:32.293107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:32.293107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:32.293249 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:32.293484 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:32.293489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:32.293613 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 04:48:32.293708 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:48:32.293839 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260928 04:48:32.294625 2999 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42583
I20260928 04:48:32.294718 2999 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:32.294867 2999 heartbeater.cc:511] Master 127.0.98.62:42583 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:48:32.295348 2808 ts_manager.cc:194] Registered new tserver with Master: 134fc76632d6411e97d99940089517ff (127.0.98.1:33651)
I20260928 04:48:32.295909 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:48:32.309957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42583
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:32.405669 3004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:32.405963 3004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:32.406006 3004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:32.407222 3004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:32.407295 3004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:32.407341 3004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:32.408942 3004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:32.409224 3004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:32.409503 3004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:32.414290 3009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:32.414319 3010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:32.414409 3004 server_base.cc:1034] running on GCE node
W20260928 04:48:32.414512 3012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:32.414692 3004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:32.414908 3004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:32.416065 3004 hybrid_clock.cc:648] HybridClock initialized: now 1790570912416059 us; error 34 us; skew 500 ppm
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:32.418516 3004 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:48:32.419008 3004 webserver.cc:460] Webserver started at http://127.0.98.2:45655/ using document root <none> and password file <none>
I20260928 04:48:32.419258 3004 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:32.419301 3004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:32.419442 3004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:32.420473 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "5b17cf88560a4da8aa2c7e704276112c"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "9e4b773697318d43081a5e4e2844608b"
server_key_iv: "c09d7061699ddf597d27315cdef43241"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.420832 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5b17cf88560a4da8aa2c7e704276112c"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "9e4b773697318d43081a5e4e2844608b"
server_key_iv: "c09d7061699ddf597d27315cdef43241"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.422160 3004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:32.422827 3019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:32.422987 3004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:32.423080 3004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "5b17cf88560a4da8aa2c7e704276112c"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "9e4b773697318d43081a5e4e2844608b"
server_key_iv: "c09d7061699ddf597d27315cdef43241"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.423156 3004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:32.436261 3004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:32.438030 3004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:32.438190 3004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:32.438400 3004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:32.438686 3004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:32.438736 3004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:32.438776 3004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:32.438802 3004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:32.444895 3004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38057
I20260928 04:48:32.444944 3132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38057 every 8 connection(s)
I20260928 04:48:32.445261 3004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:32.453994 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:32.445951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45323 (local address 127.0.98.62:42583)
0928 04:48:32.446014 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:32.446016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:32.446196 (+ 180us) server_negotiation.cc:407] Connection header received
0928 04:48:32.447145 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:32.447146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:32.447155 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:32.447215 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:32.447766 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.448235 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.449284 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.449458 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.449510 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:32.451472 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:32.451474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:32.451474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:32.451480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:32.452847 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:32.453217 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:32.453218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:32.453218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:32.453274 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:32.453467 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:32.453467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:32.453468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:32.453570 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:32.453620 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:32.453877 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 04:48:32.453931 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:48:32.454190 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:32.445925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42583 (local address 127.0.98.2:45323)
0928 04:48:32.446066 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:32.446104 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:32.447055 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:32.447268 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:32.447271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:32.447507 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:32.447696 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:32.447699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.448308 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.448309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:32.449138 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:32.449140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.449524 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.449525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:32.449625 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:32.450044 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:32.450051 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:32.451388 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:32.452911 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:32.452915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:32.452917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:32.453114 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:32.453327 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:32.453328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:32.453328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:32.453415 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:32.453622 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:32.453625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:32.453770 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 04:48:32.453850 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 04:48:32.453959 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260928 04:48:32.454355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 3004
I20260928 04:48:32.454444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:32.454638 392 external_mini_cluster.cc:1428] Setting key b4615d1cbd1ba76922307464026e4aa1
I20260928 04:48:32.454803 3133 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42583
I20260928 04:48:32.454914 3133 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:32.455080 3133 heartbeater.cc:511] Master 127.0.98.62:42583 requested a full tablet report, sending...
I20260928 04:48:32.455403 2808 ts_manager.cc:194] Registered new tserver with Master: 5b17cf88560a4da8aa2c7e704276112c (127.0.98.2:38057)
I20260928 04:48:32.455746 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45323
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260928 04:48:32.707938 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.253s user 0.000s sys 0.003s
I20260928 04:48:32.708281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42583
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:32.784171 3138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:32.784376 3138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:32.784410 3138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:32.785547 3138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:32.785600 3138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:32.785642 3138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:32.787029 3138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:32.787288 3138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:32.787492 3138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:32.792164 3146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:32.792164 3143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:32.792232 3144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:32.792315 3138 server_base.cc:1034] running on GCE node
I20260928 04:48:32.792598 3138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:32.792784 3138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:32.793912 3138 hybrid_clock.cc:648] HybridClock initialized: now 1790570912793898 us; error 30 us; skew 500 ppm
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:32.796263 3138 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:48:32.796662 3138 webserver.cc:460] Webserver started at http://127.0.98.3:34659/ using document root <none> and password file <none>
I20260928 04:48:32.796830 3138 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:32.796867 3138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:32.796945 3138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:32.797869 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "4dc9d584605c44e2aacf0e2c9a7aef62"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "4629a21419a581f9f29b6759c3269ca7"
server_key_iv: "1a861b3eb6d9d06a0a1439a2c5e7dc3a"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.798144 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4dc9d584605c44e2aacf0e2c9a7aef62"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "4629a21419a581f9f29b6759c3269ca7"
server_key_iv: "1a861b3eb6d9d06a0a1439a2c5e7dc3a"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.799255 3138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:32.799908 3153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:32.800071 3138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:32.800158 3138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "4dc9d584605c44e2aacf0e2c9a7aef62"
format_stamp: "Formatted at 2026-09-28 04:48:32 on dist-test-slave-jccs"
server_key: "4629a21419a581f9f29b6759c3269ca7"
server_key_iv: "1a861b3eb6d9d06a0a1439a2c5e7dc3a"
server_key_version: "encryptionkey@0"
I20260928 04:48:32.800235 3138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:32.813868 3138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:32.815968 3138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:32.816108 3138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:32.816315 3138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:32.816599 3138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:32.816658 3138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:32.816695 3138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:32.816723 3138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:32.822494 3138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33639
I20260928 04:48:32.822535 3266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33639 every 8 connection(s)
I20260928 04:48:32.822813 3138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:32.831096 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:32.823489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40169 (local address 127.0.98.62:42583)
0928 04:48:32.823562 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:32.823564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:32.823720 (+ 156us) server_negotiation.cc:407] Connection header received
0928 04:48:32.824590 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:32.824591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:32.824601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:32.824664 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:32.825212 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.825674 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.826584 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.826768 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.826867 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:32.828602 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:32.828603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:32.828604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:32.828611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:32.829783 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:32.830168 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:32.830170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:32.830170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:32.830237 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:32.830457 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:32.830458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:32.830459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:32.830584 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:32.830668 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:32.830934 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 04:48:32.831005 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:48:32.831274 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:32.823425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42583 (local address 127.0.98.3:40169)
0928 04:48:32.823611 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:32.823644 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:32.824494 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:32.824720 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:32.824723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:32.824924 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:32.825091 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:32.825095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.825763 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.825764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:32.826497 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:32.826498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:32.826878 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:32.826879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:32.827001 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:32.827367 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:32.827374 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:32.828517 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:32.829876 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:32.829882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:32.829883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:32.830095 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:32.830298 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:32.830299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:32.830299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:32.830391 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:32.830662 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:32.830667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:32.830811 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 04:48:32.830912 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 04:48:32.831062 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260928 04:48:32.831724 3267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42583
I20260928 04:48:32.831812 3267 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:32.831976 3267 heartbeater.cc:511] Master 127.0.98.62:42583 requested a full tablet report, sending...
I20260928 04:48:32.832315 2808 ts_manager.cc:194] Registered new tserver with Master: 4dc9d584605c44e2aacf0e2c9a7aef62 (127.0.98.3:33639)
I20260928 04:48:32.832420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 3138
I20260928 04:48:32.832520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:32.832679 392 external_mini_cluster.cc:1428] Setting key 6c03883e338fabd3d8b14d73e90cb68d
I20260928 04:48:32.832746 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40169
I20260928 04:48:32.833472 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 04:48:32.843016 2847 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:32.842236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.0.98.62:42583)
0928 04:48:32.842366 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:32.842367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:32.842396 (+ 29us) server_negotiation.cc:407] Connection header received
0928 04:48:32.842447 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:32.842447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:32.842454 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:32.842498 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:32.842939 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47614: BlockingRecv error: recv got EOF from 127.0.0.1:47614 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 04:48:32.843834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 2870
I20260928 04:48:32.848380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 3004
I20260928 04:48:32.852317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 3138
I20260928 04:48:32.856050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 2776
2026-09-28T04:48:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1954 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[3279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[3279](info): set up 2 sockets
Sep 28 04:48:32 dist-test-slave-jccs krb5kdc[3279](info): commencing operation
krb5kdc: starting...
W20260928 04:48:33.856851 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:33 dist-test-slave-jccs krb5kdc[3279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:33.965777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:34.041565 3292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:34.041749 3292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:34.041769 3292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:34.042832 3292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:34.042867 3292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:34.042881 3292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:34.042914 3292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:34.042938 3292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:34.044281 3292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:34.044500 3292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:34.044700 3292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:34.049301 3300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:34.049320 3292 server_base.cc:1034] running on GCE node
W20260928 04:48:34.049346 3298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:34.049306 3297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:34.049746 3292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:34.050050 3292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:34.051230 3292 hybrid_clock.cc:648] HybridClock initialized: now 1790570914051208 us; error 33 us; skew 500 ppm
Sep 28 04:48:34 dist-test-slave-jccs krb5kdc[3279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:34.053699 3292 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:34.054123 3292 webserver.cc:460] Webserver started at http://127.0.98.62:39807/ using document root <none> and password file <none>
I20260928 04:48:34.054282 3292 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:34.054319 3292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:34.054399 3292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:34.055276 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d"
format_stamp: "Formatted at 2026-09-28 04:48:34 on dist-test-slave-jccs"
server_key: "392069e3ed8d81b0d37129b5f6692dc7"
server_key_iv: "322bc8691991106aca872737db58a550"
server_key_version: "encryptionkey@0"
I20260928 04:48:34.055543 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d"
format_stamp: "Formatted at 2026-09-28 04:48:34 on dist-test-slave-jccs"
server_key: "392069e3ed8d81b0d37129b5f6692dc7"
server_key_iv: "322bc8691991106aca872737db58a550"
server_key_version: "encryptionkey@0"
I20260928 04:48:34.056653 3292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:34.057324 3307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:34.057533 3292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:48:34.057627 3292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d"
format_stamp: "Formatted at 2026-09-28 04:48:34 on dist-test-slave-jccs"
server_key: "392069e3ed8d81b0d37129b5f6692dc7"
server_key_iv: "322bc8691991106aca872737db58a550"
server_key_version: "encryptionkey@0"
I20260928 04:48:34.057693 3292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:34.082665 3292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:34.084291 3292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:34.084442 3292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:34.087922 3292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46463
I20260928 04:48:34.087984 3359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46463 every 8 connection(s)
I20260928 04:48:34.088266 3292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:34.089116 3360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:34.089974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 3292
I20260928 04:48:34.090044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:34.090210 392 external_mini_cluster.cc:1428] Setting key 130a43c9c7a7ab9af95b039fdc4307ed
I20260928 04:48:34.092167 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d: Bootstrap starting.
I20260928 04:48:34.092737 3360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:34.092999 3360 log.cc:824] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:34.093691 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d: No bootstrap required, opened a new log
I20260928 04:48:34.095988 3360 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } }
I20260928 04:48:34.096129 3360 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:34.096151 3360 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0e4ac6af8b472e9fdd121ea0d47f1d, State: Initialized, Role: FOLLOWER
I20260928 04:48:34.096287 3360 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } }
I20260928 04:48:34.096357 3360 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:34.096379 3360 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:34.096439 3360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [term 0 FOLLOWER]: Advancing to term 1
Sep 28 04:48:34 dist-test-slave-jccs krb5kdc[3279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:34.097061 3360 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } }
I20260928 04:48:34.097162 3360 leader_election.cc:302] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0e4ac6af8b472e9fdd121ea0d47f1d; no voters:
I20260928 04:48:34.097342 3360 leader_election.cc:288] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:34.097409 3365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:34.097558 3365 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [term 1 LEADER]: Becoming Leader. State: Replica: fb0e4ac6af8b472e9fdd121ea0d47f1d, State: Running, Role: LEADER
I20260928 04:48:34.097582 3360 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:34.097683 3365 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } }
I20260928 04:48:34.097976 3367 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } } }
I20260928 04:48:34.098152 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0e4ac6af8b472e9fdd121ea0d47f1d. Latest consensus state: current_term: 1 leader_uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0e4ac6af8b472e9fdd121ea0d47f1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46463 } } }
I20260928 04:48:34.098719 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:34.098707 3367 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:34.099026 3380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:34.099640 3380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:34.100682 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:34.090652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.0.98.62:46463)
0928 04:48:34.090813 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:34.090820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:34.090844 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:48:34.091215 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:34.091221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:34.091260 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:34.091366 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:34.092155 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:34.093045 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:34.094007 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:34.094370 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:34.094457 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:34.097068 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:34.097072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:34.097072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:34.097083 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:34.099228 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:34.099557 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:34.099560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:34.099561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:34.099649 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:34.099874 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:34.099875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:34.099875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:34.100091 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:34.100234 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:34.100360 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 04:48:34.100489 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260928 04:48:34.101246 3380 catalog_manager.cc:1347] Generated new cluster ID: 5793c469ab1247739760fdc9d978d75e
I20260928 04:48:34.101305 3380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:34.101511 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:48:34.121464 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:34.117761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.0.98.62:46463)
0928 04:48:34.117801 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:34.117802 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:34.117937 (+ 135us) server_negotiation.cc:407] Connection header received
0928 04:48:34.117970 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:34.117971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:34.117981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:34.118038 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:34.118359 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:34.119071 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:34.119945 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:34.120237 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:34.120304 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:34.120989 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:34.120990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:34.120991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:34.121215 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:34.121233 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:34.121240 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:34.121294 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:34.121359 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 04:48:34.121384 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:48:34.130111 3380 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:34.130882 3380 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:34.144972 3380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb0e4ac6af8b472e9fdd121ea0d47f1d: Generated new TSK 0
I20260928 04:48:34.145164 3380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:48:34.152650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 3292
2026-09-28T04:48:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1294 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:34 dist-test-slave-jccs krb5kdc[3393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:34 dist-test-slave-jccs krb5kdc[3393](info): set up 2 sockets
Sep 28 04:48:34 dist-test-slave-jccs krb5kdc[3393](info): commencing operation
krb5kdc: starting...
W20260928 04:48:35.148844 392 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:35.257061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39401
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:35.332998 3406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:35.333202 3406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:35.333251 3406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:35.334388 3406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:35.334427 3406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:35.334448 3406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:35.334499 3406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:35.334517 3406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:35.335918 3406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:35.336139 3406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:35.336354 3406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:35.340977 3411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:35.341032 3412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:35.341107 3406 server_base.cc:1034] running on GCE node
W20260928 04:48:35.341050 3414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:35.341475 3406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:35.341755 3406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:35.342927 3406 hybrid_clock.cc:648] HybridClock initialized: now 1790570915342914 us; error 27 us; skew 500 ppm
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:35.345085 3406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:35.345523 3406 webserver.cc:460] Webserver started at http://127.0.98.62:38483/ using document root <none> and password file <none>
I20260928 04:48:35.345737 3406 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:35.345798 3406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:35.345971 3406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:35.346922 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "4606299af4b94613869005b4d145776f"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "47d865ce578d11be8ceaa104e08d7241"
server_key_iv: "bd2cc58d8453fa400b91b7cbf901a4cf"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.347270 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "4606299af4b94613869005b4d145776f"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "47d865ce578d11be8ceaa104e08d7241"
server_key_iv: "bd2cc58d8453fa400b91b7cbf901a4cf"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.348548 3406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:35.349237 3421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.349432 3406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.349519 3406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "4606299af4b94613869005b4d145776f"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "47d865ce578d11be8ceaa104e08d7241"
server_key_iv: "bd2cc58d8453fa400b91b7cbf901a4cf"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.349601 3406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:35.356494 3406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:35.358137 3406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:35.358271 3406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:35.361964 3406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39401
I20260928 04:48:35.361985 3473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39401 every 8 connection(s)
I20260928 04:48:35.362294 3406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:35.363164 3474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:35.365360 3474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f: Bootstrap starting.
I20260928 04:48:35.365861 3474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:35.366115 3474 log.cc:824] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:35.366698 3474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f: No bootstrap required, opened a new log
I20260928 04:48:35.368155 3474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606299af4b94613869005b4d145776f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } }
I20260928 04:48:35.368286 3474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:35.368305 3474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4606299af4b94613869005b4d145776f, State: Initialized, Role: FOLLOWER
I20260928 04:48:35.368418 3474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606299af4b94613869005b4d145776f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } }
I20260928 04:48:35.368492 3474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:35.368513 3474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:35.368538 3474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:35.369282 3474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606299af4b94613869005b4d145776f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } }
I20260928 04:48:35.369365 3474 leader_election.cc:302] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4606299af4b94613869005b4d145776f; no voters:
I20260928 04:48:35.369472 3474 leader_election.cc:288] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:35.369540 3477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:35.369666 3477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [term 1 LEADER]: Becoming Leader. State: Replica: 4606299af4b94613869005b4d145776f, State: Running, Role: LEADER
I20260928 04:48:35.369701 3474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:35.369793 3477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606299af4b94613869005b4d145776f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } }
I20260928 04:48:35.370178 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4606299af4b94613869005b4d145776f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606299af4b94613869005b4d145776f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } } }
I20260928 04:48:35.370216 3479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4606299af4b94613869005b4d145776f. Latest consensus state: current_term: 1 leader_uuid: "4606299af4b94613869005b4d145776f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4606299af4b94613869005b4d145776f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39401 } } }
I20260928 04:48:35.370383 3479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:35.370368 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:35.370662 3486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:35.370949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 3406
I20260928 04:48:35.371043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:35.371145 3486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:35.371224 392 external_mini_cluster.cc:1428] Setting key 6df24fe47da73b94a6c08b2ecaa7586b
I20260928 04:48:35.372737 3486 catalog_manager.cc:1347] Generated new cluster ID: b4d2a874b3364648ba3c407ac9128126
I20260928 04:48:35.372786 3486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:35.380270 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:35.371641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45516 (local address 127.0.98.62:39401)
0928 04:48:35.371829 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:35.371849 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:35.371897 (+ 48us) server_negotiation.cc:407] Connection header received
0928 04:48:35.371998 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:35.372003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:35.372042 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:35.372163 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:35.372889 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.373796 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.374544 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.374977 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.375081 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:35.377076 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:35.377080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:35.377080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:35.377091 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:35.378691 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:35.379042 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:35.379045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:35.379045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:35.379170 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:35.379407 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:35.379408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:35.379409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:35.379640 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:35.379808 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:35.379880 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 04:48:35.380124 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260928 04:48:35.411181 3486 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:35.411731 3486 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:35.416254 3486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4606299af4b94613869005b4d145776f: Generated new TSK 0
I20260928 04:48:35.416435 3486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:48:35.443481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:35.552459 3500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:35.552652 3500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:35.552675 3500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:35.554060 3500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:35.554160 3500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:35.554219 3500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:35.555998 3500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:35.556253 3500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:35.556510 3500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:35.561633 3506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:35.561743 3508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:35.561784 3505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:35.561795 3500 server_base.cc:1034] running on GCE node
I20260928 04:48:35.562032 3500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:35.562263 3500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:35.563422 3500 hybrid_clock.cc:648] HybridClock initialized: now 1790570915563415 us; error 45 us; skew 500 ppm
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:35.565941 3500 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:48:35.566388 3500 webserver.cc:460] Webserver started at http://127.0.98.1:33783/ using document root <none> and password file <none>
I20260928 04:48:35.566625 3500 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:35.566694 3500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:35.566847 3500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:35.567899 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "63aca8a0d0084da7b3f12591fd69fd63"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "628abb8618293869ed213f497347cb18"
server_key_iv: "d63baa66c8eb085b499938c780bc285f"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.568315 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "63aca8a0d0084da7b3f12591fd69fd63"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "628abb8618293869ed213f497347cb18"
server_key_iv: "d63baa66c8eb085b499938c780bc285f"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.569796 3500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 04:48:35.570660 3515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.570837 3500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.570945 3500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "63aca8a0d0084da7b3f12591fd69fd63"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "628abb8618293869ed213f497347cb18"
server_key_iv: "d63baa66c8eb085b499938c780bc285f"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.571033 3500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:35.595222 3500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:35.597678 3500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:35.597855 3500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:35.598079 3500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:35.598373 3500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:35.598424 3500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.598465 3500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:35.598492 3500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.603916 3500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33623
I20260928 04:48:35.603964 3628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33623 every 8 connection(s)
I20260928 04:48:35.604251 3500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:48:35.608850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 3500
I20260928 04:48:35.608938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:35.609127 392 external_mini_cluster.cc:1428] Setting key 48a091ac32031243c70b1563596de132
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:35.613548 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:35.604930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60435 (local address 127.0.98.62:39401)
0928 04:48:35.605004 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:35.605006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:35.605158 (+ 152us) server_negotiation.cc:407] Connection header received
0928 04:48:35.605656 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:35.605657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:35.605669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:35.605730 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:35.606584 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.607054 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.607951 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.608134 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.608182 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:35.610713 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:35.610715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:35.610715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:35.610721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:35.612075 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:35.612509 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:35.612510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:35.612510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:35.612578 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:35.612846 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:35.612847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:35.612848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:35.613000 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:35.613057 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:35.613377 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 04:48:35.613455 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 04:48:35.613718 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:35.604887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39401 (local address 127.0.98.1:60435)
0928 04:48:35.605032 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:35.605071 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:35.605577 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:35.605804 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:35.605807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:35.606336 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:35.606511 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:35.606515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.607125 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.607126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:35.607883 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:35.607884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.608205 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.608206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:35.608302 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:35.608815 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:35.608830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:35.610630 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:35.612170 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:35.612176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:35.612178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:35.612393 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:35.612668 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:35.612668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:35.612669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:35.612780 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:35.613059 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:35.613066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:35.613240 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 04:48:35.613357 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 04:48:35.613505 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260928 04:48:35.614212 3629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39401
I20260928 04:48:35.614300 3629 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:35.614490 3629 heartbeater.cc:511] Master 127.0.98.62:39401 requested a full tablet report, sending...
I20260928 04:48:35.614943 3438 ts_manager.cc:194] Registered new tserver with Master: 63aca8a0d0084da7b3f12591fd69fd63 (127.0.98.1:33623)
I20260928 04:48:35.615536 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60435
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:48:35.632344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:35.708185 3634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:35.708395 3634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:35.708432 3634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:35.709767 3634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:35.709879 3634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:35.709926 3634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:35.711400 3634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:35.711654 3634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:35.711879 3634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:35.716440 3639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:35.716447 3642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:35.716440 3640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:35.716706 3634 server_base.cc:1034] running on GCE node
I20260928 04:48:35.716912 3634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:35.717147 3634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:35.718357 3634 hybrid_clock.cc:648] HybridClock initialized: now 1790570915718337 us; error 39 us; skew 500 ppm
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:35.720638 3634 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:48:35.721014 3634 webserver.cc:460] Webserver started at http://127.0.98.2:37877/ using document root <none> and password file <none>
I20260928 04:48:35.721256 3634 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:35.721318 3634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:35.721451 3634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:35.722373 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "23f571a002b848c68142729c5ef37bfc"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "8c2a33b5abbd65df49259d3eafdc8186"
server_key_iv: "a81724d57fd2d91056d5e844627bdb29"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.722718 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "23f571a002b848c68142729c5ef37bfc"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "8c2a33b5abbd65df49259d3eafdc8186"
server_key_iv: "a81724d57fd2d91056d5e844627bdb29"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.723913 3634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:48:35.724560 3649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.724745 3634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:35.724812 3634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "23f571a002b848c68142729c5ef37bfc"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "8c2a33b5abbd65df49259d3eafdc8186"
server_key_iv: "a81724d57fd2d91056d5e844627bdb29"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.724922 3634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:35.748031 3634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:35.749825 3634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:35.749985 3634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:35.750185 3634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:35.750470 3634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:35.750506 3634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.750547 3634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:35.750573 3634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.756042 3634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38995
I20260928 04:48:35.756112 3762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38995 every 8 connection(s)
I20260928 04:48:35.756383 3634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:48:35.756654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 3634
I20260928 04:48:35.756750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:35.756950 392 external_mini_cluster.cc:1428] Setting key a600199f81974ff5630fb71485f6abac
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:48:35.766263 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:35.757142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39749 (local address 127.0.98.62:39401)
0928 04:48:35.757235 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:35.757237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:35.757424 (+ 187us) server_negotiation.cc:407] Connection header received
0928 04:48:35.757970 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:35.757971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:35.757980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:35.758023 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:35.759210 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.760098 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.761120 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.761471 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.761559 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:35.763544 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:35.763545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:35.763546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:35.763553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:35.765058 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:35.765458 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:35.765459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:35.765459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:35.765512 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:35.765719 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:35.765719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:35.765719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:35.765803 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:35.765847 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:35.766152 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 04:48:35.766208 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:48:35.766413 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:35.757070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39401 (local address 127.0.98.2:39749)
0928 04:48:35.757307 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:35.757340 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:35.757865 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:35.758092 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:35.758096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:35.758846 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:35.759110 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:35.759117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.760191 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.760192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:35.761017 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:35.761020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.761561 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.761562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:35.761656 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:35.762156 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:35.762164 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:35.763453 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:35.765149 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:35.765153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:35.765155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:35.765390 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:35.765572 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:35.765572 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:35.765572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:35.765669 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:35.765877 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:35.765881 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:35.766047 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 04:48:35.766126 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 04:48:35.766227 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260928 04:48:35.766950 3763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39401
I20260928 04:48:35.767057 3763 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:35.767217 3763 heartbeater.cc:511] Master 127.0.98.62:39401 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 04:48:35.767643 3438 ts_manager.cc:194] Registered new tserver with Master: 23f571a002b848c68142729c5ef37bfc (127.0.98.2:38995)
I20260928 04:48:35.768018 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:48:35.780084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:35.854373 3768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:35.854542 3768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:35.854562 3768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:35.855661 3768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:35.855700 3768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:35.855746 3768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:35.857030 3768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:35.857261 3768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:35.857463 3768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:35.862004 3774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:35.862078 3773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:35.862032 3776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:35.862094 3768 server_base.cc:1034] running on GCE node
I20260928 04:48:35.862335 3768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:35.862507 3768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:35.863708 3768 hybrid_clock.cc:648] HybridClock initialized: now 1790570915863684 us; error 42 us; skew 500 ppm
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:35.866021 3768 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:48:35.866447 3768 webserver.cc:460] Webserver started at http://127.0.98.3:36789/ using document root <none> and password file <none>
I20260928 04:48:35.866657 3768 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:35.866719 3768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:35.866859 3768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:35.867781 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "97083f4dcb30481b9257a6eab5bdd244"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "2602d1901417751d81c508ab5ba8b170"
server_key_iv: "67561a3c70fc28ff61692af8bf1314c8"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.868103 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "97083f4dcb30481b9257a6eab5bdd244"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "2602d1901417751d81c508ab5ba8b170"
server_key_iv: "67561a3c70fc28ff61692af8bf1314c8"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.869251 3768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:48:35.869989 3783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.870133 3768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:35.870226 3768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "97083f4dcb30481b9257a6eab5bdd244"
format_stamp: "Formatted at 2026-09-28 04:48:35 on dist-test-slave-jccs"
server_key: "2602d1901417751d81c508ab5ba8b170"
server_key_iv: "67561a3c70fc28ff61692af8bf1314c8"
server_key_version: "encryptionkey@0"
I20260928 04:48:35.870273 3768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:35.878760 3768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:35.880724 3768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:35.880821 3768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:35.880976 3768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:35.881255 3768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:35.881307 3768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.881328 3768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:35.881340 3768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:35.886727 3768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42311
I20260928 04:48:35.886796 3896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42311 every 8 connection(s)
I20260928 04:48:35.887090 3768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:35.893939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 3768
I20260928 04:48:35.894007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:35.894187 392 external_mini_cluster.cc:1428] Setting key 0c28fbba3e3d5f37abef228171829b5a
I20260928 04:48:35.895555 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:35.887728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49825 (local address 127.0.98.62:39401)
0928 04:48:35.887787 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:35.887789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:35.887990 (+ 201us) server_negotiation.cc:407] Connection header received
0928 04:48:35.888411 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:35.888412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:35.888422 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:35.888502 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:35.889408 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.889888 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.890749 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.890909 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.890954 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:35.892909 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:35.892910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:35.892910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:35.892916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:35.894193 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:35.894654 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:35.894655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:35.894655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:35.894716 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:35.894962 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:35.894963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:35.894963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:35.895090 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:35.895160 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:35.895405 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 04:48:35.895475 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:48:35.895702 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:35.887714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39401 (local address 127.0.98.3:49825)
0928 04:48:35.887874 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:35.887911 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:35.888336 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:35.888550 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:35.888553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:35.889094 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:35.889340 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:35.889344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.889965 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.889965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:35.890685 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:35.890686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.890990 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.890991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:35.891078 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:35.891559 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:35.891568 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:35.892832 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:35.894289 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:35.894295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:35.894297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:35.894561 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:35.894783 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:35.894784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:35.894785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:35.894892 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:35.895152 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:35.895158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:35.895278 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 04:48:35.895369 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 04:48:35.895507 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260928 04:48:35.896066 3897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39401
I20260928 04:48:35.896134 3897 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:35.896317 3897 heartbeater.cc:511] Master 127.0.98.62:39401 requested a full tablet report, sending...
I20260928 04:48:35.896636 3438 ts_manager.cc:194] Registered new tserver with Master: 97083f4dcb30481b9257a6eab5bdd244 (127.0.98.3:42311)
I20260928 04:48:35.896982 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49825
I20260928 04:48:35.905797 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570915, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:35.926419 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:35.920034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.0.98.62:39401)
0928 04:48:35.920082 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:35.920084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:35.920143 (+ 59us) server_negotiation.cc:407] Connection header received
0928 04:48:35.920217 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:35.920218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:35.920227 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:35.920272 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:35.920799 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.921325 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.922012 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:35.922183 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:35.922230 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:35.924335 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:35.924336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:35.924337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:35.924342 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:35.925537 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:35.925815 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:35.925815 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:35.925816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:35.925861 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:35.926018 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:35.926019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:35.926019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:35.926138 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:35.926204 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:35.926284 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 04:48:35.926334 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 04:48:35.926678 3438 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45522
I20260928 04:48:35.927111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 3500
I20260928 04:48:35.932204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 3634
I20260928 04:48:35.936419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 3768
I20260928 04:48:35.940383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 3406
2026-09-28T04:48:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1789 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3909](info): set up 2 sockets
Sep 28 04:48:35 dist-test-slave-jccs krb5kdc[3909](info): commencing operation
krb5kdc: starting...
W20260928 04:48:36.944941 392 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:37.053946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36683
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:37.130414 3922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:37.130667 3922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:37.130703 3922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:37.131842 3922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:37.131908 3922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:37.131932 3922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:37.131960 3922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:37.131992 3922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:37.133471 3922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:37.133729 3922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:37.133945 3922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:37.138556 3927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:37.138533 3928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:37.138561 3922 server_base.cc:1034] running on GCE node
W20260928 04:48:37.138533 3930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:37.139001 3922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:37.139276 3922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:37.140436 3922 hybrid_clock.cc:648] HybridClock initialized: now 1790570917140428 us; error 33 us; skew 500 ppm
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:37.147693 3922 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:37.148077 3922 webserver.cc:460] Webserver started at http://127.0.98.62:36773/ using document root <none> and password file <none>
I20260928 04:48:37.148285 3922 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:37.148326 3922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:37.148456 3922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:37.149463 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "1dbebc00e79b4b50a60355e71be7fb71"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "821ee9200f64b60fc1d5b976518720bb"
server_key_iv: "38ef95d4314d5acd6302af905bf53f29"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.149781 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "1dbebc00e79b4b50a60355e71be7fb71"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "821ee9200f64b60fc1d5b976518720bb"
server_key_iv: "38ef95d4314d5acd6302af905bf53f29"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.151019 3922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:37.151731 3937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.151916 3922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:37.152060 3922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "1dbebc00e79b4b50a60355e71be7fb71"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "821ee9200f64b60fc1d5b976518720bb"
server_key_iv: "38ef95d4314d5acd6302af905bf53f29"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.152151 3922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:37.162325 3922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:37.163938 3922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:37.164122 3922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:37.167641 3922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36683
I20260928 04:48:37.167672 3989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36683 every 8 connection(s)
I20260928 04:48:37.167977 3922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:37.168246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 3922
I20260928 04:48:37.168354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:37.168586 392 external_mini_cluster.cc:1428] Setting key a834c30a254e9c25ebff935c7bad0a91
I20260928 04:48:37.168959 3990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:37.172070 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71: Bootstrap starting.
I20260928 04:48:37.172677 3990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:37.172967 3990 log.cc:824] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:37.173712 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71: No bootstrap required, opened a new log
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:37.175866 3990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbebc00e79b4b50a60355e71be7fb71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } }
I20260928 04:48:37.176025 3990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:37.176051 3990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbebc00e79b4b50a60355e71be7fb71, State: Initialized, Role: FOLLOWER
I20260928 04:48:37.176189 3990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbebc00e79b4b50a60355e71be7fb71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } }
I20260928 04:48:37.176281 3990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:37.176326 3990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:37.176373 3990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:37.176855 3990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbebc00e79b4b50a60355e71be7fb71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } }
I20260928 04:48:37.176967 3990 leader_election.cc:302] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dbebc00e79b4b50a60355e71be7fb71; no voters:
I20260928 04:48:37.177167 3990 leader_election.cc:288] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:37.177222 3995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:37.177358 3995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [term 1 LEADER]: Becoming Leader. State: Replica: 1dbebc00e79b4b50a60355e71be7fb71, State: Running, Role: LEADER
I20260928 04:48:37.177443 3990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:37.177549 3995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbebc00e79b4b50a60355e71be7fb71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } }
I20260928 04:48:37.177899 3997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dbebc00e79b4b50a60355e71be7fb71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbebc00e79b4b50a60355e71be7fb71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } } }
I20260928 04:48:37.178009 3997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:37.177892 3995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dbebc00e79b4b50a60355e71be7fb71. Latest consensus state: current_term: 1 leader_uuid: "1dbebc00e79b4b50a60355e71be7fb71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbebc00e79b4b50a60355e71be7fb71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36683 } } }
I20260928 04:48:37.178486 3995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:37.178717 4004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:37.179461 4004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:37.179682 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:37.169071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.0.98.62:36683)
0928 04:48:37.169276 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:37.169284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:37.169312 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:48:37.169781 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:37.169789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:37.169848 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:37.169989 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:37.170546 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.171885 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.172659 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.173301 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.173388 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:37.175930 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:37.175933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:37.175933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:37.175943 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:37.177954 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.178375 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.178377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.178377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.178450 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.178694 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.178695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.178695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.178992 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:37.179100 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:37.179329 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 04:48:37.179456 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
I20260928 04:48:37.180857 4004 catalog_manager.cc:1347] Generated new cluster ID: c1696f79b1f642d394374c20d8e733ba
I20260928 04:48:37.180905 4004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:37.200476 4004 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:37.200950 4004 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:37.203809 4004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dbebc00e79b4b50a60355e71be7fb71: Generated new TSK 0
I20260928 04:48:37.203971 4004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:48:37.231686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36683
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:37.314625 4016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:37.314812 4016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:37.314833 4016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:37.316023 4016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:37.316061 4016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:37.316098 4016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:37.317525 4016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:37.317693 4016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:37.317922 4016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:37.322621 4024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:37.322744 4022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:37.322602 4021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:37.322831 4016 server_base.cc:1034] running on GCE node
I20260928 04:48:37.323063 4016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:37.323240 4016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:37.324374 4016 hybrid_clock.cc:648] HybridClock initialized: now 1790570917324364 us; error 29 us; skew 500 ppm
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:37.326818 4016 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:48:37.327232 4016 webserver.cc:460] Webserver started at http://127.0.98.1:36849/ using document root <none> and password file <none>
I20260928 04:48:37.327443 4016 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:37.327507 4016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:37.327648 4016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:37.328595 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "32cd3d217c6244528ff5a11808e9f052"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "9488dd9728ae1876cad1628e94548911"
server_key_iv: "18351b74025a57cad1f88b747653c4d4"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.328954 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32cd3d217c6244528ff5a11808e9f052"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "9488dd9728ae1876cad1628e94548911"
server_key_iv: "18351b74025a57cad1f88b747653c4d4"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.330250 4016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:48:37.330896 4031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.331069 4016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:37.331162 4016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "32cd3d217c6244528ff5a11808e9f052"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "9488dd9728ae1876cad1628e94548911"
server_key_iv: "18351b74025a57cad1f88b747653c4d4"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.331244 4016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:37.348949 4016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:37.350564 4016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:37.350706 4016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:37.350920 4016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:37.351184 4016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:37.351249 4016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.351285 4016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:37.351310 4016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.357239 4016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44337
I20260928 04:48:37.357291 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44337 every 8 connection(s)
I20260928 04:48:37.357549 4016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:37.365157 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:37.358128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48197 (local address 127.0.98.62:36683)
0928 04:48:37.358197 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:37.358200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:37.358324 (+ 124us) server_negotiation.cc:407] Connection header received
0928 04:48:37.359187 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:37.359188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:37.359197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:37.359241 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:37.359725 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.360156 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.361027 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.361204 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.361245 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:37.363058 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:37.363059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:37.363059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:37.363065 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:37.364150 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.364480 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.364480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.364481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.364526 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.364689 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.364690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.364690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.364785 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:37.364832 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:37.365040 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 04:48:37.365097 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:48:37.365386 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:37.358087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36683 (local address 127.0.98.1:48197)
0928 04:48:37.358219 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:37.358251 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:37.359113 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:37.359294 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:37.359297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:37.359502 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:37.359661 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:37.359664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.360220 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.360221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:37.360964 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:37.360966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.361259 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.361260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:37.361343 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:37.361713 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:37.361719 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:37.362973 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:37.364215 (+ 1242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:37.364220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:37.364222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:37.364418 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:37.364566 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:37.364567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:37.364567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:37.364643 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:37.364838 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:37.364842 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:37.364939 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 04:48:37.365015 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 04:48:37.365117 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
I20260928 04:48:37.365881 4145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36683
I20260928 04:48:37.365969 4145 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:37.366132 4145 heartbeater.cc:511] Master 127.0.98.62:36683 requested a full tablet report, sending...
I20260928 04:48:37.366513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 4016
I20260928 04:48:37.366614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:37.366649 3954 ts_manager.cc:194] Registered new tserver with Master: 32cd3d217c6244528ff5a11808e9f052 (127.0.98.1:44337)
I20260928 04:48:37.366796 392 external_mini_cluster.cc:1428] Setting key bea2f7bd0284325ce0fb48a4be7ea33b
I20260928 04:48:37.367487 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48197
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:48:37.389825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36683
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:37.465540 4150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:37.465760 4150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:37.465798 4150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:37.466884 4150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:37.466948 4150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:37.466990 4150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:37.468371 4150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:37.468626 4150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:37.468833 4150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:37.473433 4156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:37.473453 4155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:37.473490 4158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:37.473690 4150 server_base.cc:1034] running on GCE node
I20260928 04:48:37.473989 4150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:37.474241 4150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:37.475401 4150 hybrid_clock.cc:648] HybridClock initialized: now 1790570917475393 us; error 27 us; skew 500 ppm
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:37.477869 4150 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:48:37.478327 4150 webserver.cc:460] Webserver started at http://127.0.98.2:37781/ using document root <none> and password file <none>
I20260928 04:48:37.478628 4150 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:37.478693 4150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:37.478844 4150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:37.479800 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "576ae0e280b342838dda9ea4fcde8a72"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "2fd21077b7b194b9149525a717bd2c9e"
server_key_iv: "637eef8ad99a56b12ff1dc7d24107f47"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.480120 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "576ae0e280b342838dda9ea4fcde8a72"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "2fd21077b7b194b9149525a717bd2c9e"
server_key_iv: "637eef8ad99a56b12ff1dc7d24107f47"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.481307 4150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:37.481976 4165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.482141 4150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:37.482237 4150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "576ae0e280b342838dda9ea4fcde8a72"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "2fd21077b7b194b9149525a717bd2c9e"
server_key_iv: "637eef8ad99a56b12ff1dc7d24107f47"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.482319 4150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:37.490571 4150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:37.492185 4150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:37.492338 4150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:37.492532 4150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:37.492818 4150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:37.492877 4150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.492913 4150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:37.492938 4150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.498606 4150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41805
I20260928 04:48:37.498652 4278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41805 every 8 connection(s)
I20260928 04:48:37.498924 4150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:48:37.503732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 4150
I20260928 04:48:37.503808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:37.503970 392 external_mini_cluster.cc:1428] Setting key 05f83a5d9d9bbe933ebf0f8d3d9706b4
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:37.507747 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:37.499604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49617 (local address 127.0.98.62:36683)
0928 04:48:37.499678 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:37.499680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:37.499831 (+ 151us) server_negotiation.cc:407] Connection header received
0928 04:48:37.500678 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:37.500679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:37.500688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:37.500748 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:37.501346 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.501818 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.502710 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.502894 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.502939 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:37.505028 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:37.505029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:37.505029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:37.505036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:37.506474 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.506866 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.506867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.506868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.506925 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.507161 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.507162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.507163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.507267 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:37.507337 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:37.507582 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 04:48:37.507660 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:48:37.507972 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:37.499563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36683 (local address 127.0.98.2:49617)
0928 04:48:37.499723 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:37.499755 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:37.500595 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:37.500799 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:37.500803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:37.501029 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:37.501268 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:37.501273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.501886 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.501887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:37.502643 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:37.502644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.502956 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.502957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:37.503042 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:37.503449 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:37.503456 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:37.504938 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:37.506570 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:37.506576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:37.506578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:37.506780 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:37.506995 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:37.506996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:37.506997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:37.507097 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:37.507330 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:37.507335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:37.507462 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 04:48:37.507563 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 04:48:37.507705 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260928 04:48:37.508348 4279 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36683
I20260928 04:48:37.508428 4279 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:37.508606 4279 heartbeater.cc:511] Master 127.0.98.62:36683 requested a full tablet report, sending...
I20260928 04:48:37.508949 3954 ts_manager.cc:194] Registered new tserver with Master: 576ae0e280b342838dda9ea4fcde8a72 (127.0.98.2:41805)
I20260928 04:48:37.509383 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49617
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:48:37.527565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36683
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:37.605638 4284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:37.605837 4284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:37.605861 4284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:37.607108 4284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:37.607151 4284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:37.607177 4284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:37.608567 4284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:37.608724 4284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:37.608892 4284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:37.613490 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:37.613620 4284 server_base.cc:1034] running on GCE node
W20260928 04:48:37.613517 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:37.613517 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:37.613907 4284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:37.614144 4284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:37.615319 4284 hybrid_clock.cc:648] HybridClock initialized: now 1790570917615307 us; error 26 us; skew 500 ppm
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:37.617694 4284 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:48:37.618126 4284 webserver.cc:460] Webserver started at http://127.0.98.3:37655/ using document root <none> and password file <none>
I20260928 04:48:37.618331 4284 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:37.618396 4284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:37.618537 4284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:37.619482 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "d44e3f28ecd4455eaf77b2250c76d263"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "40c0298c8ef50f2dce742ffe8e26bbf4"
server_key_iv: "40eab2a623e232225242e6941a03db71"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.619814 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d44e3f28ecd4455eaf77b2250c76d263"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "40c0298c8ef50f2dce742ffe8e26bbf4"
server_key_iv: "40eab2a623e232225242e6941a03db71"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.621013 4284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:48:37.621771 4299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.621938 4284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.622032 4284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "d44e3f28ecd4455eaf77b2250c76d263"
format_stamp: "Formatted at 2026-09-28 04:48:37 on dist-test-slave-jccs"
server_key: "40c0298c8ef50f2dce742ffe8e26bbf4"
server_key_iv: "40eab2a623e232225242e6941a03db71"
server_key_version: "encryptionkey@0"
I20260928 04:48:37.622118 4284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:37.641299 4284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:37.643667 4284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:37.643776 4284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:37.643961 4284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:37.644307 4284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:37.644343 4284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.644366 4284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:37.644414 4284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:37.649905 4284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38725
I20260928 04:48:37.649979 4412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38725 every 8 connection(s)
I20260928 04:48:37.650285 4284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:48:37.651634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 4284
I20260928 04:48:37.651779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:37.651965 392 external_mini_cluster.cc:1428] Setting key 6aea03a6a4df2507e45e05d4a40c91de
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:37.659121 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:37.650996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36129 (local address 127.0.98.62:36683)
0928 04:48:37.651039 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:37.651042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:37.651260 (+ 218us) server_negotiation.cc:407] Connection header received
0928 04:48:37.652242 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:37.652243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:37.652254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:37.652311 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:37.653091 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.653644 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.654725 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.654960 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.655013 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:37.656902 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:37.656903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:37.656903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:37.656909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:37.658074 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.658408 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.658408 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.658409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.658456 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.658646 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.658647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.658647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.658737 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:37.658783 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:37.658998 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 04:48:37.659057 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:48:37.659230 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:37.650994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36683 (local address 127.0.98.3:36129)
0928 04:48:37.651144 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:37.651175 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:37.652138 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:37.652398 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:37.652403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:37.652706 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:37.653015 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:37.653022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.653714 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.653715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:37.654661 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:37.654663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.655028 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.655029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:37.655136 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:37.655485 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:37.655492 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:37.656824 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:37.658136 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:37.658141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:37.658142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:37.658345 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:37.658510 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:37.658511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:37.658511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:37.658595 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:37.658787 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:37.658790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:37.658886 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 04:48:37.658960 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 04:48:37.659061 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260928 04:48:37.659657 4413 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36683
I20260928 04:48:37.659727 4413 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:37.659849 4413 heartbeater.cc:511] Master 127.0.98.62:36683 requested a full tablet report, sending...
I20260928 04:48:37.660171 3954 ts_manager.cc:194] Registered new tserver with Master: d44e3f28ecd4455eaf77b2250c76d263 (127.0.98.3:38725)
I20260928 04:48:37.660514 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36129
I20260928 04:48:37.663712 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:48:37.686028 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:37.678167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.0.98.1:44337)
0928 04:48:37.678327 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:37.678332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:37.678349 (+ 17us) server_negotiation.cc:407] Connection header received
0928 04:48:37.678374 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:37.678376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:37.678389 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:37.678452 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:37.679193 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.679886 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.680578 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.680878 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.680929 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:37.683134 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:37.683143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:37.683147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:37.683154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:37.684720 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.685144 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.685147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.685148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.685259 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.685456 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.685457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.685458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.685591 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:37.685746 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:37.685799 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 04:48:37.685899 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
Sep 28 04:48:37 dist-test-slave-jccs krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:37.692999 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:37.687202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.0.98.62:36683)
0928 04:48:37.687244 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:37.687245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:37.687289 (+ 44us) server_negotiation.cc:407] Connection header received
0928 04:48:37.687346 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:37.687347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:37.687355 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:37.687398 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:37.687784 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.688239 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.688835 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:37.689000 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:37.689047 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:37.690988 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:37.690989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:37.690990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:37.690996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:37.692102 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.692388 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.692389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.692389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.692435 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:37.692544 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:37.692544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:37.692545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:37.692663 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:37.692730 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:37.692811 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 04:48:37.692874 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 04:48:37.693276 3954 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52506
I20260928 04:48:37.693585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 4016
I20260928 04:48:37.698758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 4150
I20260928 04:48:37.703231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 4284
I20260928 04:48:37.707665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 3922
2026-09-28T04:48:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1767 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:38 dist-test-slave-jccs krb5kdc[4425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:38 dist-test-slave-jccs krb5kdc[4425](info): set up 2 sockets
Sep 28 04:48:38 dist-test-slave-jccs krb5kdc[4425](info): commencing operation
krb5kdc: starting...
W20260928 04:48:38.989162 392 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:39 dist-test-slave-jccs krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:39.099819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39999
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:39.176365 4438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:39.176599 4438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:39.176640 4438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:39.177810 4438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:39.177879 4438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:39.177903 4438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:39.177924 4438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:39.177953 4438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:39.179397 4438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:39.179654 4438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:39.179867 4438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:39.184487 4443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:39.184495 4444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:39.184494 4446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:39.184576 4438 server_base.cc:1034] running on GCE node
I20260928 04:48:39.184899 4438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:39.185233 4438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:39.186384 4438 hybrid_clock.cc:648] HybridClock initialized: now 1790570919186361 us; error 41 us; skew 500 ppm
Sep 28 04:48:39 dist-test-slave-jccs krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:39.188759 4438 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:39.189131 4438 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:39.189230 4438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:39.189394 4438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:39.190450 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "92e9a6521eb64f9d833f1fdb4bb80e03"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "b0de346fd8c1e91d4801d5b09bf7f86e"
server_key_iv: "4c3333c2c2b33c841ef3f78808304a4e"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.190794 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "92e9a6521eb64f9d833f1fdb4bb80e03"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "b0de346fd8c1e91d4801d5b09bf7f86e"
server_key_iv: "4c3333c2c2b33c841ef3f78808304a4e"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.192013 4438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:48:39.192664 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.192843 4438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.192906 4438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "92e9a6521eb64f9d833f1fdb4bb80e03"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "b0de346fd8c1e91d4801d5b09bf7f86e"
server_key_iv: "4c3333c2c2b33c841ef3f78808304a4e"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.192987 4438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:39.209982 4438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:39.212255 4438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:39.212419 4438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:39.216074 4438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39999
I20260928 04:48:39.216113 4504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39999 every 8 connection(s)
I20260928 04:48:39.216416 4438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:39.217362 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:39.219470 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03: Bootstrap starting.
I20260928 04:48:39.219992 4505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:39.220223 4505 log.cc:824] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:39.220758 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03: No bootstrap required, opened a new log
I20260928 04:48:39.222296 4505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e9a6521eb64f9d833f1fdb4bb80e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39999 } }
I20260928 04:48:39.222424 4505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:39.222446 4505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e9a6521eb64f9d833f1fdb4bb80e03, State: Initialized, Role: FOLLOWER
I20260928 04:48:39.222551 4505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e9a6521eb64f9d833f1fdb4bb80e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39999 } }
I20260928 04:48:39.222628 4505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:39.222662 4505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:39.222710 4505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:39.223470 4505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e9a6521eb64f9d833f1fdb4bb80e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39999 } }
I20260928 04:48:39.223593 4505 leader_election.cc:302] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92e9a6521eb64f9d833f1fdb4bb80e03; no voters:
I20260928 04:48:39.223682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 4438
I20260928 04:48:39.223744 4505 leader_election.cc:288] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:39.223783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:39.223820 4508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:39.223981 392 external_mini_cluster.cc:1428] Setting key 9af41e45f2ebc337622bff9ab1ddd244
I20260928 04:48:39.224004 4508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [term 1 LEADER]: Becoming Leader. State: Replica: 92e9a6521eb64f9d833f1fdb4bb80e03, State: Running, Role: LEADER
I20260928 04:48:39.224049 4505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:39.224073 4508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e9a6521eb64f9d833f1fdb4bb80e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39999 } }
I20260928 04:48:39.224375 4509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92e9a6521eb64f9d833f1fdb4bb80e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e9a6521eb64f9d833f1fdb4bb80e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39999 } } }
I20260928 04:48:39.224448 4509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:39.224588 4510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e9a6521eb64f9d833f1fdb4bb80e03. Latest consensus state: current_term: 1 leader_uuid: "92e9a6521eb64f9d833f1fdb4bb80e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e9a6521eb64f9d833f1fdb4bb80e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39999 } } }
I20260928 04:48:39.224900 4510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:39.224937 4514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:39.225492 4514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:39.227059 4514 catalog_manager.cc:1347] Generated new cluster ID: 1a1e546f044c4078a5ecb43c55462624
I20260928 04:48:39.227108 4514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:48:39 dist-test-slave-jccs krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:39.235688 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.224712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.0.98.62:39999)
0928 04:48:39.225273 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:39.225283 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:39.225317 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:48:39.225482 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:39.225489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:39.225544 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:39.226271 (+ 727us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:39.226367 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.227736 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.228814 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.229489 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.229613 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.232279 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:39.232283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:39.232283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:39.232294 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:39.234183 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:39.234568 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:39.234571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:39.234571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:39.234672 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:39.234881 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:39.234881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:39.234882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:39.235147 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:39.235278 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:39.235383 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 04:48:39.235500 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":58,"threads_started":1}
I20260928 04:48:39.242435 4514 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:39.242935 4514 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:39.247501 4514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03: Generated new TSK 0
I20260928 04:48:39.247658 4514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:48:39.276420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39999
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:39.354126 4531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:39.354314 4531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:39.354337 4531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:39.355509 4531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:39.355551 4531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:39.355613 4531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:39.356949 4531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:39.357213 4531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:39.357441 4531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:39.361879 4536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:39.362018 4539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:39.361876 4537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:39.362066 4531 server_base.cc:1034] running on GCE node
I20260928 04:48:39.362303 4531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:39.362533 4531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:39.363690 4531 hybrid_clock.cc:648] HybridClock initialized: now 1790570919363687 us; error 36 us; skew 500 ppm
Sep 28 04:48:39 dist-test-slave-jccs krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:39.366163 4531 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:48:39.366405 4531 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:39.366441 4531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:39.366528 4531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:39.367374 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "febf98cb42a541073f0396aec8f2de8e"
server_key_iv: "df5e3e3559b996a257e742bb7b5d4e3c"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.367700 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "febf98cb42a541073f0396aec8f2de8e"
server_key_iv: "df5e3e3559b996a257e742bb7b5d4e3c"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.369359 4531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:39.370070 4545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.370211 4531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:39.370286 4531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "febf98cb42a541073f0396aec8f2de8e"
server_key_iv: "df5e3e3559b996a257e742bb7b5d4e3c"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.370368 4531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:39.404953 4531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:39.407408 4531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:39.407569 4531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:39.407814 4531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:39.408133 4531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:39.408169 4531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.408210 4531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:39.408241 4531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.413546 4531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36717
I20260928 04:48:39.413583 4658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36717 every 8 connection(s)
I20260928 04:48:39.413841 4531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:48:39 dist-test-slave-jccs krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:39.421140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 4531
I20260928 04:48:39.421271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:39.421438 392 external_mini_cluster.cc:1428] Setting key d495b2e1688f6b2d1529bc84e2d8f4a4
I20260928 04:48:39.422574 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.414587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53607 (local address 127.0.98.62:39999)
0928 04:48:39.414658 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:39.414660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:39.414810 (+ 150us) server_negotiation.cc:407] Connection header received
0928 04:48:39.415239 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:39.415240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:39.415253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:39.415313 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:39.416142 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.416590 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.417559 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.417756 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.417807 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.419666 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:39.419668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:39.419668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:39.419677 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:39.421062 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:39.421569 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:39.421570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:39.421571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:39.421629 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:39.421870 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:39.421878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:39.421878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:39.421989 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:39.422086 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:39.422385 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 04:48:39.422489 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:48:39.422731 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.414540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39999 (local address 127.0.98.1:53607)
0928 04:48:39.414701 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:39.414734 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:39.415166 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:39.415378 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:39.415381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:39.415620 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:39.416073 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.416077 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.416660 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.416660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.417487 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.417489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.417822 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.417822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.417931 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.418321 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:39.418328 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:39.419588 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:39.421196 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:39.421202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:39.421204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:39.421479 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:39.421695 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:39.421696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:39.421696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:39.421804 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:39.422078 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:39.422084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:39.422228 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 04:48:39.422351 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 04:48:39.422509 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260928 04:48:39.423173 4659 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39999
I20260928 04:48:39.423274 4659 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:39.423429 4659 heartbeater.cc:511] Master 127.0.98.62:39999 requested a full tablet report, sending...
I20260928 04:48:39.423943 4469 ts_manager.cc:194] Registered new tserver with Master: a6f7d92cc6ae4c51b99ce6af8aa841c6 (127.0.98.1:36717)
I20260928 04:48:39.424616 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53607
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:48:39.445326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39999
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:39.520867 4664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:39.521054 4664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:39.521077 4664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:39.522255 4664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:39.522295 4664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:39.522322 4664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:39.523759 4664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:39.523931 4664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:39.524092 4664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:39.528787 4670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:39.528784 4669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:39.528782 4672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:39.529039 4664 server_base.cc:1034] running on GCE node
I20260928 04:48:39.529286 4664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:39.529464 4664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:39.530625 4664 hybrid_clock.cc:648] HybridClock initialized: now 1790570919530609 us; error 31 us; skew 500 ppm
Sep 28 04:48:39 dist-test-slave-jccs krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:39.532992 4664 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:48:39.533367 4664 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:39.533416 4664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:39.533566 4664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:39.534539 4664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "504460b751104285bbabb70e8d15a413"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "53eb647ccf3e8c1770ba651c3178179f"
server_key_iv: "5e976336a1bac6a5888f00bbb71d82fc"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.534866 4664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "504460b751104285bbabb70e8d15a413"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "53eb647ccf3e8c1770ba651c3178179f"
server_key_iv: "5e976336a1bac6a5888f00bbb71d82fc"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.536017 4664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:48:39.536685 4678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.536852 4664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:39.536944 4664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "504460b751104285bbabb70e8d15a413"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "53eb647ccf3e8c1770ba651c3178179f"
server_key_iv: "5e976336a1bac6a5888f00bbb71d82fc"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.537029 4664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:39.553381 4664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:39.555049 4664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:39.555194 4664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:39.555403 4664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:39.555686 4664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:39.555734 4664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.555766 4664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:39.555789 4664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.561385 4664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42089
I20260928 04:48:39.561435 4791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42089 every 8 connection(s)
I20260928 04:48:39.561691 4664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:48:39 dist-test-slave-jccs krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:39.569233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 4664
I20260928 04:48:39.569341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:39.569550 392 external_mini_cluster.cc:1428] Setting key 79c14e56e514a63d5a904f361b523db5
I20260928 04:48:39.570812 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.562355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55741 (local address 127.0.98.62:39999)
0928 04:48:39.562432 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:39.562434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:39.562572 (+ 138us) server_negotiation.cc:407] Connection header received
0928 04:48:39.562991 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:39.562992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:39.563002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:39.563052 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:39.563884 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.564372 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.565342 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.565522 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.565573 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.567394 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:39.567395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:39.567396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:39.567402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:39.568903 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:39.569526 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:39.569527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:39.569528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:39.569586 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:39.569902 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:39.569903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:39.569904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:39.570062 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:39.570121 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:39.570568 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 04:48:39.570678 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:48:39.570879 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.562299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39999 (local address 127.0.98.2:55741)
0928 04:48:39.562465 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:39.562496 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:39.562916 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:39.563103 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:39.563106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:39.563360 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:39.563815 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.563819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.564445 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.564446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.565267 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.565269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.565588 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.565589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.565678 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.566060 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:39.566067 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:39.567315 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:39.568990 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:39.568995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:39.568997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:39.569385 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:39.569662 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:39.569663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:39.569664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:39.569798 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:39.570152 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:39.570157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:39.570360 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 04:48:39.570518 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 04:48:39.570675 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260928 04:48:39.572531 4792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39999
I20260928 04:48:39.572633 4792 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:39.572813 4792 heartbeater.cc:511] Master 127.0.98.62:39999 requested a full tablet report, sending...
I20260928 04:48:39.573153 4469 ts_manager.cc:194] Registered new tserver with Master: 504460b751104285bbabb70e8d15a413 (127.0.98.2:42089)
I20260928 04:48:39.573570 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55741
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:48:39.593920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39999
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:39.670475 4797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:39.670655 4797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:39.670675 4797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:39.671687 4797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:39.671720 4797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:39.671746 4797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:39.673211 4797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:39.673385 4797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:39.673537 4797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:39.678223 4805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:39.678218 4802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:39.678382 4797 server_base.cc:1034] running on GCE node
W20260928 04:48:39.678434 4803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:39.678622 4797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:39.678833 4797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:39.679963 4797 hybrid_clock.cc:648] HybridClock initialized: now 1790570919679954 us; error 26 us; skew 500 ppm
Sep 28 04:48:39 dist-test-slave-jccs krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:39.682437 4797 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:48:39.682773 4797 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:39.682839 4797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:39.682976 4797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:39.683990 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "f4e438c7eba84bbe90f972d53be1133f"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "75a4b9ea580601e59b42ea7f597dd909"
server_key_iv: "e455b41528b53b83cc32e61de8ac08f5"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.684346 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f4e438c7eba84bbe90f972d53be1133f"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "75a4b9ea580601e59b42ea7f597dd909"
server_key_iv: "e455b41528b53b83cc32e61de8ac08f5"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.685616 4797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:39.686271 4811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.686436 4797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.686548 4797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "f4e438c7eba84bbe90f972d53be1133f"
format_stamp: "Formatted at 2026-09-28 04:48:39 on dist-test-slave-jccs"
server_key: "75a4b9ea580601e59b42ea7f597dd909"
server_key_iv: "e455b41528b53b83cc32e61de8ac08f5"
server_key_version: "encryptionkey@0"
I20260928 04:48:39.686622 4797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:39.711542 4797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:39.714025 4797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:39.714195 4797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:39.714464 4797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:39.714803 4797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:39.714874 4797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.714923 4797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:39.714955 4797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:39.720415 4797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38147
I20260928 04:48:39.720466 4924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38147 every 8 connection(s)
I20260928 04:48:39.720770 4797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:48:39 dist-test-slave-jccs krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:39.728410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 4797
I20260928 04:48:39.728531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:39.728742 392 external_mini_cluster.cc:1428] Setting key 5f8e93c0722c2bcfb168c0557357f323
I20260928 04:48:39.729475 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.721460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35743 (local address 127.0.98.62:39999)
0928 04:48:39.721522 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:39.721524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:39.721705 (+ 181us) server_negotiation.cc:407] Connection header received
0928 04:48:39.722174 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:39.722175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:39.722184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:39.722240 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:39.723058 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.723542 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.724590 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.724772 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.724828 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.726754 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:39.726756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:39.726756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:39.726762 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:39.728122 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:39.728445 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:39.728446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:39.728447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:39.728507 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:39.728816 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:39.728817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:39.728817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:39.728944 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:39.729006 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:39.729320 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 04:48:39.729395 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:48:39.729696 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.721447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39999 (local address 127.0.98.3:35743)
0928 04:48:39.721592 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:39.721629 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:39.722070 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:39.722289 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:39.722292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:39.722526 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:39.722987 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.722991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.723619 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.723620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.724518 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.724525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.724841 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.724842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.724930 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.725458 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:39.725466 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:39.726693 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:39.728186 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:39.728189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:39.728191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:39.728367 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:39.728598 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:39.728600 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:39.728600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:39.728737 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:39.729013 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:39.729019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:39.729165 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 04:48:39.729298 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 04:48:39.729461 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260928 04:48:39.730089 4925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39999
I20260928 04:48:39.730190 4925 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:39.730326 4925 heartbeater.cc:511] Master 127.0.98.62:39999 requested a full tablet report, sending...
I20260928 04:48:39.730674 4469 ts_manager.cc:194] Registered new tserver with Master: f4e438c7eba84bbe90f972d53be1133f (127.0.98.3:38147)
I20260928 04:48:39.731033 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35743
I20260928 04:48:39.740278 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:48:39.746855 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.741529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.0.98.62:39999)
0928 04:48:39.741605 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:39.741606 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:39.741646 (+ 40us) server_negotiation.cc:407] Connection header received
0928 04:48:39.741695 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:39.741696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:39.741703 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:39.741746 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:39.742258 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.742696 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.743415 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.743578 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.743622 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.744775 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:39.744776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:39.744776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:39.744781 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:39.745943 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:39.746235 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:39.746236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:39.746236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:39.746298 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:39.746465 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:39.746466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:39.746466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:39.746581 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:39.746651 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:39.746731 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 04:48:39.746778 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 04:48:39.748448 4469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:39.750018 4469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:39.756313 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.752127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42089 (local address 127.0.0.1:36640)
0928 04:48:39.752268 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:39.752305 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:39.752369 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:39.752651 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:39.752653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:39.752655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:39.752699 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.752700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.753852 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.753852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.755106 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.755108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.756036 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.756037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.756125 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.756138 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 04:48:39.756208 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:48:39.756244 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":51,"threads_started":1}
I20260928 04:48:39.756369 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.752223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.0.98.2:42089)
0928 04:48:39.752363 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:39.752368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:39.752379 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:48:39.752467 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:39.752469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:39.752484 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:39.752556 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:39.752768 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.753743 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.755189 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.755940 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.756011 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.756064 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:39.756240 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 04:48:39.756285 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260928 04:48:39.756553 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.752314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38147 (local address 127.0.0.1:59596)
0928 04:48:39.752879 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:39.752889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:39.752935 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:39.753217 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:39.753217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:39.753219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:39.753256 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.753257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.754501 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.754502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.755505 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.755507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.756347 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.756348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.756408 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.756412 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:39.756454 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 04:48:39.756488 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":56,"threads_started":1}
I20260928 04:48:39.756688 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.752429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.0.98.3:38147)
0928 04:48:39.752561 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:39.752566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:39.752987 (+ 421us) server_negotiation.cc:407] Connection header received
0928 04:48:39.753025 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:39.753027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:39.753041 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:39.753106 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:39.753329 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.754395 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.755594 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.756260 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.756327 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.756524 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:39.756545 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 04:48:39.756574 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260928 04:48:39.757248 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.752133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36717 (local address 127.0.0.1:42922)
0928 04:48:39.752569 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:39.752581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:39.752632 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:39.752792 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:39.752793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:39.752794 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:39.752832 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.752848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.754222 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.754223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.755276 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.755278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.756981 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.756982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.757052 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.757055 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:39.757122 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 04:48:39.757159 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":86,"threads_started":1}
I20260928 04:48:39.757304 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.752444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.0.98.1:36717)
0928 04:48:39.752622 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:39.752627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:39.752638 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:48:39.752693 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:39.752695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:39.752709 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:39.752786 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:39.752934 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.754074 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.755366 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.755947 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.756037 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.756091 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:39.757160 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0928 04:48:39.757223 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260928 04:48:39.757391 4859 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf56e34a0494e86bd4e37dc0f518dfa (DEFAULT_TABLE table=test-table [id=9bcd4a40797e47999d62e98c60cb76d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:39.757395 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf56e34a0494e86bd4e37dc0f518dfa (DEFAULT_TABLE table=test-table [id=9bcd4a40797e47999d62e98c60cb76d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:39.757715 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf56e34a0494e86bd4e37dc0f518dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:39.757715 4859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf56e34a0494e86bd4e37dc0f518dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:39.757823 4593 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf56e34a0494e86bd4e37dc0f518dfa (DEFAULT_TABLE table=test-table [id=9bcd4a40797e47999d62e98c60cb76d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:39.758056 4593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf56e34a0494e86bd4e37dc0f518dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:39.760617 4942 tablet_bootstrap.cc:492] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6: Bootstrap starting.
I20260928 04:48:39.760890 4943 tablet_bootstrap.cc:492] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: Bootstrap starting.
I20260928 04:48:39.761278 4942 tablet_bootstrap.cc:654] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:39.761618 4943 tablet_bootstrap.cc:654] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:39.761893 4942 log.cc:824] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:39.762214 4943 log.cc:824] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:39.762504 4944 tablet_bootstrap.cc:492] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: Bootstrap starting.
I20260928 04:48:39.762696 4942 tablet_bootstrap.cc:492] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6: No bootstrap required, opened a new log
I20260928 04:48:39.762773 4942 ts_tablet_manager.cc:1397] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:48:39.763103 4944 tablet_bootstrap.cc:654] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:39.763654 4944 log.cc:824] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:39.764356 4944 tablet_bootstrap.cc:492] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: No bootstrap required, opened a new log
I20260928 04:48:39.764423 4944 ts_tablet_manager.cc:1397] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:39.764897 4942 raft_consensus.cc:348] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504460b751104285bbabb70e8d15a413" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42089 } } peers { permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 } } peers { permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 } }
I20260928 04:48:39.765074 4942 raft_consensus.cc:374] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:39.765112 4942 raft_consensus.cc:729] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f7d92cc6ae4c51b99ce6af8aa841c6, State: Initialized, Role: FOLLOWER
I20260928 04:48:39.765249 4942 consensus_queue.cc:260] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504460b751104285bbabb70e8d15a413" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42089 } } peers { permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 } } peers { permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 } }
I20260928 04:48:39.765511 4942 ts_tablet_manager.cc:1428] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 04:48:39.765553 4659 heartbeater.cc:503] Master 127.0.98.62:39999 was elected leader, sending a full tablet report...
I20260928 04:48:39.766217 4943 tablet_bootstrap.cc:492] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: No bootstrap required, opened a new log
I20260928 04:48:39.766307 4943 ts_tablet_manager.cc:1397] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260928 04:48:39.766472 4944 raft_consensus.cc:348] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504460b751104285bbabb70e8d15a413" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42089 } } peers { permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 } } peers { permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 } }
I20260928 04:48:39.766635 4944 raft_consensus.cc:374] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:39.766683 4944 raft_consensus.cc:729] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504460b751104285bbabb70e8d15a413, State: Initialized, Role: FOLLOWER
I20260928 04:48:39.766796 4944 consensus_queue.cc:260] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504460b751104285bbabb70e8d15a413" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42089 } } peers { permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 } } peers { permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 } }
I20260928 04:48:39.767066 4944 ts_tablet_manager.cc:1428] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:48:39.767076 4792 heartbeater.cc:503] Master 127.0.98.62:39999 was elected leader, sending a full tablet report...
I20260928 04:48:39.768309 4943 raft_consensus.cc:348] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504460b751104285bbabb70e8d15a413" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42089 } } peers { permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 } } peers { permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 } }
I20260928 04:48:39.768424 4943 raft_consensus.cc:374] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:39.768445 4943 raft_consensus.cc:729] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e438c7eba84bbe90f972d53be1133f, State: Initialized, Role: FOLLOWER
I20260928 04:48:39.768574 4943 consensus_queue.cc:260] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504460b751104285bbabb70e8d15a413" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42089 } } peers { permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 } } peers { permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 } }
I20260928 04:48:39.768826 4943 ts_tablet_manager.cc:1428] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 04:48:39.768851 4925 heartbeater.cc:503] Master 127.0.98.62:39999 was elected leader, sending a full tablet report...
W20260928 04:48:39.812208 4793 tablet.cc:2369] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:48:39.914616 4660 tablet.cc:2369] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:48:39.971361 4926 tablet.cc:2369] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:48:39.985592 4949 raft_consensus.cc:482] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:39.985751 4949 raft_consensus.cc:504] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504460b751104285bbabb70e8d15a413" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42089 } } peers { permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 } } peers { permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 } }
I20260928 04:48:39.986127 4949 leader_election.cc:288] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6f7d92cc6ae4c51b99ce6af8aa841c6 (127.0.98.1:36717), f4e438c7eba84bbe90f972d53be1133f (127.0.98.3:38147)
I20260928 04:48:39.988750 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.986273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36717 (local address 127.0.98.2:47005)
0928 04:48:39.986337 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:39.986356 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:39.986433 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:39.986606 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:39.986608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:39.986610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:39.986676 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.986678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.987300 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.987301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.988012 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.988013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.988514 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.988515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.988582 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.988585 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:39.988640 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:48:39.988678 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 04:48:39.988781 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.986348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47005 (local address 127.0.98.1:36717)
0928 04:48:39.986405 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:39.986407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:39.986416 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:39.986499 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:39.986501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:39.986514 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:39.986599 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:39.986755 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.987220 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.988098 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.988439 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.988516 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.988525 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:39.988663 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 04:48:39.988710 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:48:39.989065 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.986401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38147 (local address 127.0.98.2:59431)
0928 04:48:39.986571 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:39.986580 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:39.986663 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:39.986823 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:39.986824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:39.986825 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:39.986904 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.986905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.987483 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.987484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.988191 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:39.988192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.988775 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.988776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:39.988847 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.988851 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:39.988963 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 04:48:39.988998 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260928 04:48:39.989104 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:39.986484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59431 (local address 127.0.98.3:38147)
0928 04:48:39.986538 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:39.986540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:39.986635 (+ 95us) server_negotiation.cc:407] Connection header received
0928 04:48:39.986716 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:39.986717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:39.986727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:39.986817 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:39.986983 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.987405 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.988283 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:39.988688 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:39.988751 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:39.988759 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:39.988992 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 04:48:39.989027 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:48:39.989079 4613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf56e34a0494e86bd4e37dc0f518dfa" candidate_uuid: "504460b751104285bbabb70e8d15a413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" is_pre_election: true
I20260928 04:48:39.989243 4613 raft_consensus.cc:2454] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504460b751104285bbabb70e8d15a413 in term 0.
I20260928 04:48:39.989498 4680 leader_election.cc:302] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504460b751104285bbabb70e8d15a413, a6f7d92cc6ae4c51b99ce6af8aa841c6; no voters:
I20260928 04:48:39.989575 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf56e34a0494e86bd4e37dc0f518dfa" candidate_uuid: "504460b751104285bbabb70e8d15a413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4e438c7eba84bbe90f972d53be1133f" is_pre_election: true
I20260928 04:48:39.989634 4949 raft_consensus.cc:2781] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:39.989691 4949 raft_consensus.cc:482] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:39.989718 4949 raft_consensus.cc:3037] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:39.989737 4879 raft_consensus.cc:2454] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504460b751104285bbabb70e8d15a413 in term 0.
I20260928 04:48:39.990373 4949 raft_consensus.cc:504] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504460b751104285bbabb70e8d15a413" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42089 } } peers { permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 } } peers { permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 } }
I20260928 04:48:39.990504 4949 leader_election.cc:288] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [CANDIDATE]: Term 1 election: Requested vote from peers a6f7d92cc6ae4c51b99ce6af8aa841c6 (127.0.98.1:36717), f4e438c7eba84bbe90f972d53be1133f (127.0.98.3:38147)
I20260928 04:48:39.990667 4613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf56e34a0494e86bd4e37dc0f518dfa" candidate_uuid: "504460b751104285bbabb70e8d15a413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6"
I20260928 04:48:39.990685 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf56e34a0494e86bd4e37dc0f518dfa" candidate_uuid: "504460b751104285bbabb70e8d15a413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4e438c7eba84bbe90f972d53be1133f"
I20260928 04:48:39.990748 4879 raft_consensus.cc:3037] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:39.990748 4613 raft_consensus.cc:3037] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:39.991545 4879 raft_consensus.cc:2454] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504460b751104285bbabb70e8d15a413 in term 1.
I20260928 04:48:39.991545 4613 raft_consensus.cc:2454] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504460b751104285bbabb70e8d15a413 in term 1.
I20260928 04:48:39.991739 4680 leader_election.cc:302] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504460b751104285bbabb70e8d15a413, f4e438c7eba84bbe90f972d53be1133f; no voters:
I20260928 04:48:39.991832 4949 raft_consensus.cc:2781] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:39.991979 4949 raft_consensus.cc:686] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 1 LEADER]: Becoming Leader. State: Replica: 504460b751104285bbabb70e8d15a413, State: Running, Role: LEADER
I20260928 04:48:39.992063 4949 consensus_queue.cc:237] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504460b751104285bbabb70e8d15a413" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42089 } } peers { permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 } } peers { permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 } }
I20260928 04:48:39.992836 4469 catalog_manager.cc:5495] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 reported cstate change: term changed from 0 to 1, leader changed from <none> to 504460b751104285bbabb70e8d15a413 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "504460b751104285bbabb70e8d15a413" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "504460b751104285bbabb70e8d15a413" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 } health_report { overall_health: UNKNOWN } } }
I20260928 04:48:40.037887 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.035332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.0.98.2:42089)
0928 04:48:40.035376 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.035378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.035386 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:40.035429 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.035430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.035449 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.035526 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:40.035670 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:40.036184 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:40.037081 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:40.037291 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:40.037360 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:40.037549 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:40.037643 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:40.037727 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.037765 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.037784 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:48:40.039547 4613 raft_consensus.cc:1261] T 8bf56e34a0494e86bd4e37dc0f518dfa P a6f7d92cc6ae4c51b99ce6af8aa841c6 [term 1 FOLLOWER]: Refusing update from remote peer 504460b751104285bbabb70e8d15a413: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:40.039547 4879 raft_consensus.cc:1261] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f [term 1 FOLLOWER]: Refusing update from remote peer 504460b751104285bbabb70e8d15a413: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:40.039891 4949 consensus_queue.cc:1035] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4e438c7eba84bbe90f972d53be1133f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:40.040110 4952 consensus_queue.cc:1035] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f7d92cc6ae4c51b99ce6af8aa841c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:40.042747 4957 mvcc.cc:205] Tried to move back new op lower bound from 7334178488478920704 to 7334178488287952896. Current Snapshot: MvccSnapshot[applied={T|T < 7334178488478920704}]
I20260928 04:48:40.043228 4956 mvcc.cc:205] Tried to move back new op lower bound from 7334178488478920704 to 7334178488287952896. Current Snapshot: MvccSnapshot[applied={T|T < 7334178488478920704}]
I20260928 04:48:40.044297 4469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:40.044392 4469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:40.045125 4469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92e9a6521eb64f9d833f1fdb4bb80e03: Sending DeleteTablet for 3 replicas of tablet 8bf56e34a0494e86bd4e37dc0f518dfa
I20260928 04:48:40.045488 4859 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf56e34a0494e86bd4e37dc0f518dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59596
I20260928 04:48:40.045565 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf56e34a0494e86bd4e37dc0f518dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36640
I20260928 04:48:40.045688 4965 tablet_replica.cc:330] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: stopping tablet replica
I20260928 04:48:40.045776 4966 tablet_replica.cc:330] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: stopping tablet replica
I20260928 04:48:40.045778 4965 raft_consensus.cc:2229] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:40.045768 4593 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf56e34a0494e86bd4e37dc0f518dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42922
I20260928 04:48:40.045845 4966 raft_consensus.cc:2229] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:40.045866 4965 raft_consensus.cc:2258] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:40.045934 4966 raft_consensus.cc:2258] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:40.046033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 4531
I20260928 04:48:40.047350 4966 ts_tablet_manager.cc:1905] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:40.048703 4965 ts_tablet_manager.cc:1905] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:40.049149 4966 ts_tablet_manager.cc:1918] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:40.049232 4966 log.cc:1196] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/8bf56e34a0494e86bd4e37dc0f518dfa
I20260928 04:48:40.049458 4966 ts_tablet_manager.cc:1939] T 8bf56e34a0494e86bd4e37dc0f518dfa P 504460b751104285bbabb70e8d15a413: Deleting consensus metadata
I20260928 04:48:40.049788 4456 catalog_manager.cc:4841] TS 504460b751104285bbabb70e8d15a413 (127.0.98.2:42089): tablet 8bf56e34a0494e86bd4e37dc0f518dfa (table test-table [id=9bcd4a40797e47999d62e98c60cb76d9]) successfully deleted
I20260928 04:48:40.050261 4965 ts_tablet_manager.cc:1918] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:40.050313 4965 log.cc:1196] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/8bf56e34a0494e86bd4e37dc0f518dfa
I20260928 04:48:40.050580 4965 ts_tablet_manager.cc:1939] T 8bf56e34a0494e86bd4e37dc0f518dfa P f4e438c7eba84bbe90f972d53be1133f: Deleting consensus metadata
I20260928 04:48:40.050918 4454 catalog_manager.cc:4841] TS f4e438c7eba84bbe90f972d53be1133f (127.0.98.3:38147): tablet 8bf56e34a0494e86bd4e37dc0f518dfa (table test-table [id=9bcd4a40797e47999d62e98c60cb76d9]) successfully deleted
W20260928 04:48:40.051613 4454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36717 (error 108)
I20260928 04:48:40.051805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 4664
I20260928 04:48:40.052028 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.051914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36717 (local address 127.0.0.1:42936)
0928 04:48:40.051952 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.051978 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 04:48:40.052381 4454 catalog_manager.cc:4584] TS a6f7d92cc6ae4c51b99ce6af8aa841c6 (127.0.98.1:36717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bf56e34a0494e86bd4e37dc0f518dfa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36717: connect: Connection refused (error 111)
I20260928 04:48:40.056840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 4797
I20260928 04:48:40.061821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 4438
2026-09-28T04:48:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2353 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T04:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:40Z Disabled control of system clock
I20260928 04:48:40.087666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33597
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 04:48:40.164404 4972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:40.164619 4972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:40.164639 4972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:40.165720 4972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:40.165755 4972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:40.165766 4972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:40.165779 4972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:40.165791 4972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:40.167019 4972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:40.167191 4972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:40.167397 4972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:40.171950 4978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:40.172070 4972 server_base.cc:1034] running on GCE node
W20260928 04:48:40.171962 4977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:40.171977 4980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:40.172389 4972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:40.172667 4972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:40.173821 4972 hybrid_clock.cc:648] HybridClock initialized: now 1790570920173810 us; error 40 us; skew 500 ppm
I20260928 04:48:40.175042 4972 webserver.cc:460] Webserver started at http://127.0.98.62:42095/ using document root <none> and password file <none>
I20260928 04:48:40.175279 4972 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:40.175350 4972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:40.175498 4972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:40.176486 4972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "8f5a315916ea4a839cb5d6bc090ff660"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "d126405693a6bc0e0acc673a912984c3"
server_key_iv: "ee96ad6cfff7d7a276b7a7b26b399be8"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.176843 4972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f5a315916ea4a839cb5d6bc090ff660"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "d126405693a6bc0e0acc673a912984c3"
server_key_iv: "ee96ad6cfff7d7a276b7a7b26b399be8"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.178149 4972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:40.178838 4986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.179008 4972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:40.179101 4972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "8f5a315916ea4a839cb5d6bc090ff660"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "d126405693a6bc0e0acc673a912984c3"
server_key_iv: "ee96ad6cfff7d7a276b7a7b26b399be8"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.179181 4972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:40.182391 4972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:40.182638 4972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:40.182781 4972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:40.186489 4972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33597
I20260928 04:48:40.186515 5038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33597 every 8 connection(s)
I20260928 04:48:40.186818 4972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:40.187386 5039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:40.190019 5039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660: Bootstrap starting.
I20260928 04:48:40.190553 5039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:40.190824 5039 log.cc:824] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:40.191447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 4972
I20260928 04:48:40.191476 5039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660: No bootstrap required, opened a new log
I20260928 04:48:40.191557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:40.191720 392 external_mini_cluster.cc:1428] Setting key fb0c6a7cb98c962420e64d10bb03aee9
I20260928 04:48:40.193550 5039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5a315916ea4a839cb5d6bc090ff660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } }
I20260928 04:48:40.193711 5039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:40.193758 5039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5a315916ea4a839cb5d6bc090ff660, State: Initialized, Role: FOLLOWER
I20260928 04:48:40.193894 5039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5a315916ea4a839cb5d6bc090ff660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } }
I20260928 04:48:40.193981 5039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:40.194011 5039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:40.194056 5039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:40.194700 5039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5a315916ea4a839cb5d6bc090ff660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } }
I20260928 04:48:40.194808 5039 leader_election.cc:302] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f5a315916ea4a839cb5d6bc090ff660; no voters:
I20260928 04:48:40.194833 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.192217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.0.98.62:33597)
0928 04:48:40.192415 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.192424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.192456 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:48:40.193512 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.193521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.193584 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.193729 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.194382 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.194386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.194386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.194390 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.194432 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.194447 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.194521 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.194572 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.194624 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 04:48:40.194947 5039 leader_election.cc:288] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:40.195170 5039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:40.195225 5044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:40.195344 5044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [term 1 LEADER]: Becoming Leader. State: Replica: 8f5a315916ea4a839cb5d6bc090ff660, State: Running, Role: LEADER
I20260928 04:48:40.195443 5044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5a315916ea4a839cb5d6bc090ff660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } }
I20260928 04:48:40.195768 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f5a315916ea4a839cb5d6bc090ff660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5a315916ea4a839cb5d6bc090ff660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } } }
I20260928 04:48:40.195859 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:40.195781 5046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f5a315916ea4a839cb5d6bc090ff660. Latest consensus state: current_term: 1 leader_uuid: "8f5a315916ea4a839cb5d6bc090ff660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5a315916ea4a839cb5d6bc090ff660" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33597 } } }
I20260928 04:48:40.196034 5046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:40.196147 5053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:40.196652 5053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:40.198024 5053 catalog_manager.cc:1347] Generated new cluster ID: d8fe02c58dbd45f696466cf8b900312a
I20260928 04:48:40.198087 5053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:40.216135 5053 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:40.216301 5053 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:40.224148 5053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660: Generated new TSK 0
I20260928 04:48:40.224334 5053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:48:40.229373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33597
--builtin_ntp_servers=127.0.98.20:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 04:48:40.309887 5063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:40.310081 5063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:40.310102 5063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:40.311224 5063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:40.311261 5063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:40.311286 5063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:40.312575 5063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:40.312796 5063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:40.313083 5063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:40.317849 5069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:40.317939 5071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:40.317888 5068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:40.318135 5063 server_base.cc:1034] running on GCE node
I20260928 04:48:40.318377 5063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:40.318602 5063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:40.319766 5063 hybrid_clock.cc:648] HybridClock initialized: now 1790570920319747 us; error 43 us; skew 500 ppm
I20260928 04:48:40.321089 5063 webserver.cc:460] Webserver started at http://127.0.98.1:39599/ using document root <none> and password file <none>
I20260928 04:48:40.321338 5063 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:40.321393 5063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:40.321509 5063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:40.322742 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "a4ad341a6166506d137e582c0ee5da8d"
server_key_iv: "4dd03a446c45209d3e5c3ed7a5f0dc39"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.323134 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "a4ad341a6166506d137e582c0ee5da8d"
server_key_iv: "4dd03a446c45209d3e5c3ed7a5f0dc39"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.324724 5063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:48:40.325407 5077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.325594 5063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:40.325671 5063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "a4ad341a6166506d137e582c0ee5da8d"
server_key_iv: "4dd03a446c45209d3e5c3ed7a5f0dc39"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.325791 5063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:40.328753 5063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:40.328991 5063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:40.329114 5063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:40.329332 5063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:40.329604 5063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:40.329633 5063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.329680 5063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:40.329702 5063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.335326 5063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39495
I20260928 04:48:40.335371 5190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39495 every 8 connection(s)
I20260928 04:48:40.335687 5063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:48:40.338342 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.336299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42923 (local address 127.0.98.62:33597)
0928 04:48:40.336354 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.336356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.336610 (+ 254us) server_negotiation.cc:407] Connection header received
0928 04:48:40.337005 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.337006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.337011 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.337062 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.338164 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.338165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.338165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.338168 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.338180 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.338184 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.338225 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.338265 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.338277 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:48:40.338534 5193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.336297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33597 (local address 127.0.98.1:42923)
0928 04:48:40.336440 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.336481 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:40.336922 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:40.337109 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:40.337111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:40.337119 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:40.338045 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:48:40.338045 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:40.338068 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:48:40.338093 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:40.338222 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:40.338224 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:40.338226 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:48:40.338270 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 04:48:40.338329 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":76,"threads_started":1}
I20260928 04:48:40.339015 5191 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33597
I20260928 04:48:40.339102 5191 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:40.339246 5191 heartbeater.cc:511] Master 127.0.98.62:33597 requested a full tablet report, sending...
I20260928 04:48:40.339705 5003 ts_manager.cc:194] Registered new tserver with Master: 2fc1ac4d669c49a08c5d1d4f375dedbe (127.0.98.1:39495)
I20260928 04:48:40.344323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 5063
I20260928 04:48:40.344462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:40.344653 392 external_mini_cluster.cc:1428] Setting key 8e871e304b4c7a473954720624cff0a7
I20260928 04:48:40.345516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33597
--builtin_ntp_servers=127.0.98.20:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 04:48:40.421442 5194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:40.421615 5194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:40.421635 5194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:40.422688 5194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:40.422719 5194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:40.422744 5194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:40.423929 5194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:40.424082 5194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:40.424345 5194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:40.429153 5202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:40.429143 5199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:40.429267 5194 server_base.cc:1034] running on GCE node
W20260928 04:48:40.429143 5200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:40.429567 5194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:40.429803 5194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:40.430970 5194 hybrid_clock.cc:648] HybridClock initialized: now 1790570920430954 us; error 34 us; skew 500 ppm
I20260928 04:48:40.432090 5194 webserver.cc:460] Webserver started at http://127.0.98.2:33943/ using document root <none> and password file <none>
I20260928 04:48:40.432369 5194 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:40.432439 5194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:40.432607 5194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:40.433686 5194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "f6223bc066cc4de8bbef6a965a62f782"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "fd8bb962997dee47b6c8b4201241356b"
server_key_iv: "6fc589d93b601dc195407ce846e66fb9"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.434011 5194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f6223bc066cc4de8bbef6a965a62f782"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "fd8bb962997dee47b6c8b4201241356b"
server_key_iv: "6fc589d93b601dc195407ce846e66fb9"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.435148 5194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:40.435763 5208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.435932 5194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:40.436025 5194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "f6223bc066cc4de8bbef6a965a62f782"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "fd8bb962997dee47b6c8b4201241356b"
server_key_iv: "6fc589d93b601dc195407ce846e66fb9"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.436091 5194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:40.439208 5194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:40.439440 5194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:40.439566 5194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:40.439754 5194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:40.440011 5194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:40.440059 5194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.440093 5194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:40.440119 5194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.445681 5194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37971
I20260928 04:48:40.445720 5321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37971 every 8 connection(s)
I20260928 04:48:40.445992 5194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:48:40.448629 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.446633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32801 (local address 127.0.98.62:33597)
0928 04:48:40.446698 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.446700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.446905 (+ 205us) server_negotiation.cc:407] Connection header received
0928 04:48:40.447364 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.447365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.447370 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.447422 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.448396 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.448397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.448397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.448401 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.448416 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.448420 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.448465 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.448546 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.448560 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:48:40.448796 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.446615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33597 (local address 127.0.98.2:32801)
0928 04:48:40.446801 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.446829 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:40.447269 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:40.447483 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:40.447492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:40.447499 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:40.448260 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:48:40.448261 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:40.448284 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:48:40.448306 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:40.448460 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:40.448463 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:40.448466 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:40.448556 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 04:48:40.448608 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260928 04:48:40.449240 5322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33597
I20260928 04:48:40.449314 5322 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:40.449429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 5194
I20260928 04:48:40.449451 5322 heartbeater.cc:511] Master 127.0.98.62:33597 requested a full tablet report, sending...
I20260928 04:48:40.449529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:40.449712 392 external_mini_cluster.cc:1428] Setting key d7a19348b357c46d9ce29e0a386b1f41
I20260928 04:48:40.449817 5003 ts_manager.cc:194] Registered new tserver with Master: f6223bc066cc4de8bbef6a965a62f782 (127.0.98.2:37971)
I20260928 04:48:40.450484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33597
--builtin_ntp_servers=127.0.98.20:39541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 04:48:40.528025 5325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:40.528280 5325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:40.528311 5325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:40.529402 5325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:40.529449 5325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:40.529487 5325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:40.530647 5325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:40.530856 5325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:40.531088 5325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:40.535763 5333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:40.535907 5331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:40.535881 5325 server_base.cc:1034] running on GCE node
W20260928 04:48:40.536080 5330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:40.536314 5325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:40.536564 5325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:40.537695 5325 hybrid_clock.cc:648] HybridClock initialized: now 1790570920537687 us; error 36 us; skew 500 ppm
I20260928 04:48:40.539000 5325 webserver.cc:460] Webserver started at http://127.0.98.3:37827/ using document root <none> and password file <none>
I20260928 04:48:40.539258 5325 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:40.539321 5325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:40.539440 5325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:40.540601 5325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "6a568db98b1a4b92867275a75f501897"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "338d0f85eee905703cf0bd72ae25498c"
server_key_iv: "35d9abce1fe0027424af69d172e46ae4"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.540882 5325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6a568db98b1a4b92867275a75f501897"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "338d0f85eee905703cf0bd72ae25498c"
server_key_iv: "35d9abce1fe0027424af69d172e46ae4"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.542150 5325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:40.543161 5339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.543350 5325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:40.543444 5325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "6a568db98b1a4b92867275a75f501897"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "338d0f85eee905703cf0bd72ae25498c"
server_key_iv: "35d9abce1fe0027424af69d172e46ae4"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.543517 5325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:40.547922 5325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:40.548111 5325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:40.548199 5325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:40.548339 5325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:40.548534 5325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:40.548559 5325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.548578 5325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:40.548588 5325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.554157 5325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45135
I20260928 04:48:40.554190 5452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45135 every 8 connection(s)
I20260928 04:48:40.554486 5325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:48:40.557102 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.555149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37387 (local address 127.0.98.62:33597)
0928 04:48:40.555212 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.555214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.555380 (+ 166us) server_negotiation.cc:407] Connection header received
0928 04:48:40.555781 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.555782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.555787 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.555835 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.556915 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.556916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.556916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.556920 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.556934 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.556939 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.556984 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.557028 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.557054 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:48:40.557315 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.555139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33597 (local address 127.0.98.3:37387)
0928 04:48:40.555289 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.555317 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:40.555702 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:40.555882 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:40.555885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:40.555892 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:40.556784 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:48:40.556785 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:40.556809 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:48:40.556834 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:40.556981 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:40.556983 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:40.556985 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:48:40.557028 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 04:48:40.557081 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260928 04:48:40.557695 5453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33597
I20260928 04:48:40.557763 5453 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:40.557906 5453 heartbeater.cc:511] Master 127.0.98.62:33597 requested a full tablet report, sending...
I20260928 04:48:40.558210 5003 ts_manager.cc:194] Registered new tserver with Master: 6a568db98b1a4b92867275a75f501897 (127.0.98.3:45135)
I20260928 04:48:40.564706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 5325
I20260928 04:48:40.564815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:40.564992 392 external_mini_cluster.cc:1428] Setting key 19a725afc4c32f5a16da9758840f63a6
I20260928 04:48:40.565840 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:48:40.567986 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.566982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.0.98.62:33597)
0928 04:48:40.567051 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.567053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.567139 (+ 86us) server_negotiation.cc:407] Connection header received
0928 04:48:40.567226 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.567227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.567232 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.567285 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.567776 (+ 491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.567777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.567777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.567781 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.567795 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.567799 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.567852 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.567906 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.567921 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:48:40.569659 5003 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:40.571071 5003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:40.575170 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.573290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45135 (local address 127.0.0.1:48820)
0928 04:48:40.573443 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.573482 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:40.573536 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:40.574153 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:40.574158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:40.574164 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:40.574748 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:48:40.574749 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:40.574764 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:48:40.574782 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:40.574979 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:40.574985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:40.574988 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:40.575050 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 04:48:40.575064 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":54,"threads_started":1}
I20260928 04:48:40.575423 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.573485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37971 (local address 127.0.0.1:41332)
0928 04:48:40.574094 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.574106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:40.574174 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:40.574571 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:40.574572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:40.574574 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:40.575029 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:48:40.575030 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:40.575036 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:48:40.575046 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:40.575298 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:40.575299 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:40.575300 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:48:40.575339 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 04:48:40.575351 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":72,"threads_started":1}
I20260928 04:48:40.575835 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.573597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.0.98.2:37971)
0928 04:48:40.574430 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.574434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.574443 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:40.574469 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.574471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.574492 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.574568 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.575145 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.575152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.575153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.575157 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.575190 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.575214 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.575271 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.575591 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.575607 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":89,"threads_started":1}
I20260928 04:48:40.575733 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.573685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.0.98.3:45135)
0928 04:48:40.574018 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.574022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.574031 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:40.574061 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.574068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.574075 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.574131 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.574861 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.574867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.574868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.574872 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.574906 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.574924 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.574980 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.575341 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.575357 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":65,"threads_started":1}
I20260928 04:48:40.576468 5256 tablet_service.cc:1467] Processing CreateTablet for tablet 620558699df74a02bdafd99925b6e6aa (DEFAULT_TABLE table=test-table [id=7f78eab79d9c48619cbb3b667badaa39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:40.576546 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.573312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39495 (local address 127.0.0.1:53220)
0928 04:48:40.573824 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.573834 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:40.573875 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:40.575292 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:40.575293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:40.575295 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:40.575678 (+ 383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:48:40.575679 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:40.575685 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:48:40.575715 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:40.576426 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:40.576427 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:40.576427 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 04:48:40.576470 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 04:48:40.576483 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":68,"threads_started":1}
I20260928 04:48:40.576727 5256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620558699df74a02bdafd99925b6e6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:40.576618 5387 tablet_service.cc:1467] Processing CreateTablet for tablet 620558699df74a02bdafd99925b6e6aa (DEFAULT_TABLE table=test-table [id=7f78eab79d9c48619cbb3b667badaa39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:40.576934 5387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620558699df74a02bdafd99925b6e6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:40.577104 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.573689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53220 (local address 127.0.98.1:39495)
0928 04:48:40.574882 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.574886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.574895 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:40.574919 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.574921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.574929 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.575002 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.575802 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.575808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.575809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.575814 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.575851 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.576334 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.576400 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.576888 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.576910 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":91,"threads_started":1}
I20260928 04:48:40.577845 5125 tablet_service.cc:1467] Processing CreateTablet for tablet 620558699df74a02bdafd99925b6e6aa (DEFAULT_TABLE table=test-table [id=7f78eab79d9c48619cbb3b667badaa39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:40.578162 5125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620558699df74a02bdafd99925b6e6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:40.579224 5467 tablet_bootstrap.cc:492] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: Bootstrap starting.
I20260928 04:48:40.579779 5467 tablet_bootstrap.cc:654] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:40.580044 5467 log.cc:824] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:40.580081 5468 tablet_bootstrap.cc:492] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: Bootstrap starting.
I20260928 04:48:40.580657 5468 tablet_bootstrap.cc:654] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:40.580885 5468 log.cc:824] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:40.581233 5471 tablet_bootstrap.cc:492] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe: Bootstrap starting.
I20260928 04:48:40.581585 5468 tablet_bootstrap.cc:492] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: No bootstrap required, opened a new log
I20260928 04:48:40.581676 5468 ts_tablet_manager.cc:1397] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:40.581847 5471 tablet_bootstrap.cc:654] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:40.581853 5467 tablet_bootstrap.cc:492] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: No bootstrap required, opened a new log
I20260928 04:48:40.581915 5467 ts_tablet_manager.cc:1397] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 04:48:40.582201 5471 log.cc:824] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:40.582924 5471 tablet_bootstrap.cc:492] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe: No bootstrap required, opened a new log
I20260928 04:48:40.582993 5471 ts_tablet_manager.cc:1397] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:48:40.583885 5468 raft_consensus.cc:348] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 } } peers { permanent_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 } }
I20260928 04:48:40.584028 5468 raft_consensus.cc:374] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:40.584072 5468 raft_consensus.cc:729] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a568db98b1a4b92867275a75f501897, State: Initialized, Role: FOLLOWER
I20260928 04:48:40.584177 5468 consensus_queue.cc:260] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 } } peers { permanent_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 } }
I20260928 04:48:40.584445 5468 ts_tablet_manager.cc:1428] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:48:40.584450 5453 heartbeater.cc:503] Master 127.0.98.62:33597 was elected leader, sending a full tablet report...
I20260928 04:48:40.584563 5467 raft_consensus.cc:348] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 } } peers { permanent_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 } }
I20260928 04:48:40.584713 5467 raft_consensus.cc:374] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:40.584753 5467 raft_consensus.cc:729] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6223bc066cc4de8bbef6a965a62f782, State: Initialized, Role: FOLLOWER
I20260928 04:48:40.584853 5467 consensus_queue.cc:260] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 } } peers { permanent_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 } }
I20260928 04:48:40.585088 5467 ts_tablet_manager.cc:1428] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:48:40.585130 5322 heartbeater.cc:503] Master 127.0.98.62:33597 was elected leader, sending a full tablet report...
I20260928 04:48:40.586041 5471 raft_consensus.cc:348] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 } } peers { permanent_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 } }
I20260928 04:48:40.586169 5471 raft_consensus.cc:374] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:40.586189 5471 raft_consensus.cc:729] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc1ac4d669c49a08c5d1d4f375dedbe, State: Initialized, Role: FOLLOWER
I20260928 04:48:40.586275 5471 consensus_queue.cc:260] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 } } peers { permanent_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 } }
I20260928 04:48:40.586506 5471 ts_tablet_manager.cc:1428] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 04:48:40.586534 5191 heartbeater.cc:503] Master 127.0.98.62:33597 was elected leader, sending a full tablet report...
I20260928 04:48:40.663791 5475 raft_consensus.cc:482] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:40.663909 5475 raft_consensus.cc:504] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 } } peers { permanent_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 } }
I20260928 04:48:40.664297 5475 leader_election.cc:288] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a568db98b1a4b92867275a75f501897 (127.0.98.3:45135), f6223bc066cc4de8bbef6a965a62f782 (127.0.98.2:37971)
I20260928 04:48:40.665701 5193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.664528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37971 (local address 127.0.98.1:44345)
0928 04:48:40.664633 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.664646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:40.664692 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:40.664807 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:40.664807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:40.664810 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:40.665330 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:48:40.665331 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:40.665338 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:48:40.665352 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:40.665531 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:40.665532 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:40.665533 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:48:40.665597 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 04:48:40.665611 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 04:48:40.665704 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.664618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44345 (local address 127.0.98.2:37971)
0928 04:48:40.664681 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.664684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.664693 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:40.664745 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.664746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.664751 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.664803 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.665442 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.665443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.665444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.665448 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.665463 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.665468 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.665526 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.665602 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.665617 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:48:40.665944 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.664675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49527 (local address 127.0.98.3:45135)
0928 04:48:40.664718 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.664721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.664809 (+ 88us) server_negotiation.cc:407] Connection header received
0928 04:48:40.664870 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.664871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.664876 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.664950 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.665701 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.665703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.665703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.665708 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.665723 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.665727 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.665760 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.665871 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.665885 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:48:40.666038 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "620558699df74a02bdafd99925b6e6aa" candidate_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6223bc066cc4de8bbef6a965a62f782" is_pre_election: true
I20260928 04:48:40.666103 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.664529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45135 (local address 127.0.98.1:49527)
0928 04:48:40.664725 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.664734 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:40.664814 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:40.664954 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:40.664955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:40.664957 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:40.665569 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:48:40.665570 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:40.665577 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:48:40.665590 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:40.665805 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:40.665805 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:40.665806 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:48:40.666026 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 04:48:40.666041 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":13,"thread_start_us":111,"threads_started":1}
I20260928 04:48:40.666190 5276 raft_consensus.cc:2454] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc1ac4d669c49a08c5d1d4f375dedbe in term 0.
I20260928 04:48:40.666430 5078 leader_election.cc:302] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc1ac4d669c49a08c5d1d4f375dedbe, f6223bc066cc4de8bbef6a965a62f782; no voters:
I20260928 04:48:40.666419 5407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "620558699df74a02bdafd99925b6e6aa" candidate_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a568db98b1a4b92867275a75f501897" is_pre_election: true
I20260928 04:48:40.666536 5475 raft_consensus.cc:2781] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:40.666574 5407 raft_consensus.cc:2454] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc1ac4d669c49a08c5d1d4f375dedbe in term 0.
I20260928 04:48:40.666610 5475 raft_consensus.cc:482] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:40.666637 5475 raft_consensus.cc:3037] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:40.667414 5475 raft_consensus.cc:504] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 } } peers { permanent_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 } }
I20260928 04:48:40.667544 5475 leader_election.cc:288] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [CANDIDATE]: Term 1 election: Requested vote from peers 6a568db98b1a4b92867275a75f501897 (127.0.98.3:45135), f6223bc066cc4de8bbef6a965a62f782 (127.0.98.2:37971)
I20260928 04:48:40.667675 5407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "620558699df74a02bdafd99925b6e6aa" candidate_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a568db98b1a4b92867275a75f501897"
I20260928 04:48:40.667747 5407 raft_consensus.cc:3037] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:40.668406 5407 raft_consensus.cc:2454] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc1ac4d669c49a08c5d1d4f375dedbe in term 1.
I20260928 04:48:40.668519 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "620558699df74a02bdafd99925b6e6aa" candidate_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6223bc066cc4de8bbef6a965a62f782"
I20260928 04:48:40.668627 5081 leader_election.cc:302] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc1ac4d669c49a08c5d1d4f375dedbe, 6a568db98b1a4b92867275a75f501897; no voters:
I20260928 04:48:40.668622 5276 raft_consensus.cc:3037] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:40.668730 5475 raft_consensus.cc:2781] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:40.668886 5475 raft_consensus.cc:686] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [term 1 LEADER]: Becoming Leader. State: Replica: 2fc1ac4d669c49a08c5d1d4f375dedbe, State: Running, Role: LEADER
I20260928 04:48:40.668977 5475 consensus_queue.cc:237] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 } } peers { permanent_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } } peers { permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 } }
I20260928 04:48:40.669482 5276 raft_consensus.cc:2454] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc1ac4d669c49a08c5d1d4f375dedbe in term 1.
I20260928 04:48:40.669842 5002 catalog_manager.cc:5495] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fc1ac4d669c49a08c5d1d4f375dedbe (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fc1ac4d669c49a08c5d1d4f375dedbe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 } health_report { overall_health: UNKNOWN } } }
I20260928 04:48:40.691515 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.690618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.0.98.1:39495)
0928 04:48:40.690682 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.690684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.690693 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:40.690757 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.690758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.690777 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.690838 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.691325 (+ 487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.691326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.691326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.691330 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.691343 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.691347 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.691395 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.691432 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.691446 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:48:40.693295 5407 raft_consensus.cc:1261] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897 [term 1 FOLLOWER]: Refusing update from remote peer 2fc1ac4d669c49a08c5d1d4f375dedbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:40.693315 5276 raft_consensus.cc:1261] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782 [term 1 FOLLOWER]: Refusing update from remote peer 2fc1ac4d669c49a08c5d1d4f375dedbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:40.693591 5475 consensus_queue.cc:1035] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a568db98b1a4b92867275a75f501897" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:40.693753 5477 consensus_queue.cc:1035] T 620558699df74a02bdafd99925b6e6aa P 2fc1ac4d669c49a08c5d1d4f375dedbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6223bc066cc4de8bbef6a965a62f782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:40.696046 5480 mvcc.cc:205] Tried to move back new op lower bound from 7334178491156688896 to 7334178491060555776. Current Snapshot: MvccSnapshot[applied={T|T < 7334178491156688896}]
I20260928 04:48:40.696678 5482 mvcc.cc:205] Tried to move back new op lower bound from 7334178491156688896 to 7334178491060555776. Current Snapshot: MvccSnapshot[applied={T|T < 7334178491156688896 or (T in {7334178491156688896})}]
I20260928 04:48:40.696833 5481 mvcc.cc:205] Tried to move back new op lower bound from 7334178491156688896 to 7334178491060555776. Current Snapshot: MvccSnapshot[applied={T|T < 7334178491156688896 or (T in {7334178491156688896})}]
I20260928 04:48:40.698625 5003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:40.698730 5003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:40.699395 5003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f5a315916ea4a839cb5d6bc090ff660: Sending DeleteTablet for 3 replicas of tablet 620558699df74a02bdafd99925b6e6aa
I20260928 04:48:40.699698 5125 tablet_service.cc:1514] Processing DeleteTablet for tablet 620558699df74a02bdafd99925b6e6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:40 UTC) from {username='slave'} at 127.0.0.1:53220
I20260928 04:48:40.699688 5387 tablet_service.cc:1514] Processing DeleteTablet for tablet 620558699df74a02bdafd99925b6e6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:40 UTC) from {username='slave'} at 127.0.0.1:48820
I20260928 04:48:40.699756 5256 tablet_service.cc:1514] Processing DeleteTablet for tablet 620558699df74a02bdafd99925b6e6aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:40 UTC) from {username='slave'} at 127.0.0.1:41332
I20260928 04:48:40.699994 5492 tablet_replica.cc:330] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: stopping tablet replica
I20260928 04:48:40.700003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 5063
I20260928 04:48:40.700033 5490 tablet_replica.cc:330] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: stopping tablet replica
I20260928 04:48:40.700114 5490 raft_consensus.cc:2229] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:40.700201 5490 raft_consensus.cc:2258] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:40.700644 5490 ts_tablet_manager.cc:1905] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:40.700685 5492 raft_consensus.cc:2229] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:40.700760 5492 raft_consensus.cc:2258] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:40.701090 5492 ts_tablet_manager.cc:1905] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:40.702448 5492 ts_tablet_manager.cc:1918] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:40.702448 5490 ts_tablet_manager.cc:1918] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:40.702497 5492 log.cc:1196] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/620558699df74a02bdafd99925b6e6aa
I20260928 04:48:40.702510 5490 log.cc:1196] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/620558699df74a02bdafd99925b6e6aa
I20260928 04:48:40.702737 5492 ts_tablet_manager.cc:1939] T 620558699df74a02bdafd99925b6e6aa P f6223bc066cc4de8bbef6a965a62f782: Deleting consensus metadata
I20260928 04:48:40.702768 5490 ts_tablet_manager.cc:1939] T 620558699df74a02bdafd99925b6e6aa P 6a568db98b1a4b92867275a75f501897: Deleting consensus metadata
I20260928 04:48:40.703109 4987 catalog_manager.cc:4841] TS f6223bc066cc4de8bbef6a965a62f782 (127.0.98.2:37971): tablet 620558699df74a02bdafd99925b6e6aa (table test-table [id=7f78eab79d9c48619cbb3b667badaa39]) successfully deleted
I20260928 04:48:40.703109 4990 catalog_manager.cc:4841] TS 6a568db98b1a4b92867275a75f501897 (127.0.98.3:45135): tablet 620558699df74a02bdafd99925b6e6aa (table test-table [id=7f78eab79d9c48619cbb3b667badaa39]) successfully deleted
W20260928 04:48:40.706745 4989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39495 (error 108)
I20260928 04:48:40.706939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 5194
I20260928 04:48:40.707464 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.707087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39495 (local address 127.0.0.1:53240)
0928 04:48:40.707397 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.707434 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
W20260928 04:48:40.707811 4989 catalog_manager.cc:4584] TS 2fc1ac4d669c49a08c5d1d4f375dedbe (127.0.98.1:39495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 620558699df74a02bdafd99925b6e6aa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39495: connect: Connection refused (error 111)
I20260928 04:48:40.711942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 5325
I20260928 04:48:40.716194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 4972
2026-09-28T04:48:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (653 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 04:48:40.734730 392 mini_oidc.cc:152] Starting JWKS server
I20260928 04:48:40.735657 392 webserver.cc:460] Webserver started at https://127.0.0.1:45335/ using document root <none> and password file <none>
I20260928 04:48:40.735728 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 04:48:40.735859 392 webserver.cc:460] Webserver started at http://127.0.0.1:39837/ using document root <none> and password file <none>
2026-09-28T04:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:40Z Disabled control of system clock
I20260928 04:48:40.743090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45601
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45601
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45335/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/testchainca.cert with env {}
W20260928 04:48:40.818874 5499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:40.819054 5499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:40.819074 5499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:40.820137 5499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:40.820174 5499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:40.820187 5499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:40.820199 5499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:40.820211 5499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:48:40.820223 5499 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:48:40.820237 5499 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45335/jwks.json
I20260928 04:48:40.821627 5499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45335/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:40.821796 5499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:40.821936 5499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:40.826375 5507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:40.826422 5505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:40.826576 5499 server_base.cc:1034] running on GCE node
W20260928 04:48:40.826633 5504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:40.826885 5499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:40.827152 5499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:40.828321 5499 hybrid_clock.cc:648] HybridClock initialized: now 1790570920828302 us; error 36 us; skew 500 ppm
I20260928 04:48:40.829485 5499 webserver.cc:460] Webserver started at http://127.0.98.62:42397/ using document root <none> and password file <none>
I20260928 04:48:40.829707 5499 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:40.829771 5499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:40.829893 5499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:40.830812 5499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "68d596d500a44f5bba6a9466a0fc5163"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "9e6bc3185722d6d652397c517cb84565"
server_key_iv: "63ad0e5cf92a8884c6a8704bfad64caa"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.831142 5499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "68d596d500a44f5bba6a9466a0fc5163"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "9e6bc3185722d6d652397c517cb84565"
server_key_iv: "63ad0e5cf92a8884c6a8704bfad64caa"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.832304 5499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:40.832927 5513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:40.833086 5499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:40.833177 5499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "68d596d500a44f5bba6a9466a0fc5163"
format_stamp: "Formatted at 2026-09-28 04:48:40 on dist-test-slave-jccs"
server_key: "9e6bc3185722d6d652397c517cb84565"
server_key_iv: "63ad0e5cf92a8884c6a8704bfad64caa"
server_key_version: "encryptionkey@0"
I20260928 04:48:40.833285 5499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:40.852358 5499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:40.852617 5499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:40.852773 5499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:40.856330 5499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45601
I20260928 04:48:40.856366 5565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45601 every 8 connection(s)
I20260928 04:48:40.856645 5499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:40.856921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 5499
I20260928 04:48:40.857015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:40.857218 392 external_mini_cluster.cc:1428] Setting key b441e9327d08fcfc7813567b56926f4f
I20260928 04:48:40.857738 5566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:40.860447 5566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163: Bootstrap starting.
I20260928 04:48:40.861141 5566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:40.861475 5566 log.cc:824] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:40.862207 5566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163: No bootstrap required, opened a new log
I20260928 04:48:40.862484 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.857645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.0.98.62:45601)
0928 04:48:40.857853 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.857862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.857898 (+ 36us) server_negotiation.cc:407] Connection header received
0928 04:48:40.858286 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.858292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.858342 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.858442 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:40.858677 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:40.859480 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:40.860532 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:40.861013 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:40.861096 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:40.861827 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:40.861830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:40.861831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:40.862010 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:40.862038 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:40.862048 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:40.862098 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:40.862195 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 04:48:40.862273 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260928 04:48:40.863335 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:48:40.864351 5566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d596d500a44f5bba6a9466a0fc5163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45601 } }
I20260928 04:48:40.864480 5566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:40.864534 5566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d596d500a44f5bba6a9466a0fc5163, State: Initialized, Role: FOLLOWER
I20260928 04:48:40.864648 5566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d596d500a44f5bba6a9466a0fc5163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45601 } }
I20260928 04:48:40.864719 5566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:40.864761 5566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:40.864806 5566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:40.865361 5566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d596d500a44f5bba6a9466a0fc5163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45601 } }
I20260928 04:48:40.865469 5566 leader_election.cc:302] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68d596d500a44f5bba6a9466a0fc5163; no voters:
I20260928 04:48:40.865615 5566 leader_election.cc:288] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:40.865653 5571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:40.865828 5571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [term 1 LEADER]: Becoming Leader. State: Replica: 68d596d500a44f5bba6a9466a0fc5163, State: Running, Role: LEADER
I20260928 04:48:40.865885 5566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:40.865918 5571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d596d500a44f5bba6a9466a0fc5163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45601 } }
I20260928 04:48:40.866253 5572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68d596d500a44f5bba6a9466a0fc5163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d596d500a44f5bba6a9466a0fc5163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45601 } } }
I20260928 04:48:40.866353 5572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:40.866251 5573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68d596d500a44f5bba6a9466a0fc5163. Latest consensus state: current_term: 1 leader_uuid: "68d596d500a44f5bba6a9466a0fc5163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d596d500a44f5bba6a9466a0fc5163" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45601 } } }
I20260928 04:48:40.866487 5573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:40.866557 5578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:40.867216 5578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:40.868618 5578 catalog_manager.cc:1347] Generated new cluster ID: 9eb75ef9a71449bcacdae0c9f868bbfd
I20260928 04:48:40.868711 5578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:40.888286 5578 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:40.888993 5578 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:40.894343 5578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68d596d500a44f5bba6a9466a0fc5163: Generated new TSK 0
I20260928 04:48:40.894523 5578 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 04:48:40.924234 5596 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 04:48:40.924324 5596 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 04:48:40.924583 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:40.909396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.0.98.62:45601)
0928 04:48:40.909456 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:40.909459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:40.909616 (+ 157us) server_negotiation.cc:407] Connection header received
0928 04:48:40.909709 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:40.909710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:40.909720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:40.909771 (+ 51us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:48:40.909955 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:40.910431 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:40.911564 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:40.911747 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:40.911799 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:40.912008 (+ 209us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:48:40.924333 (+ 12325us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 04:48:40.924499 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 04:48:40.924657 5569 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 04:48:40.924894 5595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:40.909351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45601 (local address 127.0.0.1:59526)
0928 04:48:40.909544 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:40.909555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:40.909624 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:40.909785 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:40.909786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:40.909788 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:48:40.909862 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:40.909864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:40.910509 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:40.910510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:40.911481 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:40.911483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:40.911817 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:40.911820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:40.911885 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:40.911904 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:48:40.924509 (+ 12605us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 04:48:40.924626 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":124,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:45601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45335/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 04:48:40.925365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 5499
2026-09-28T04:48:40Z chronyd exiting
I20260928 04:48:40.960547 392 test_util.cc:175] -----------------------------------------------
I20260928 04:48:40.960626 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790570897511889-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 04:48:40.961575 392 mini_oidc.cc:152] Starting JWKS server
I20260928 04:48:40.962229 392 webserver.cc:460] Webserver started at https://127.0.0.1:46009/ using document root <none> and password file <none>
I20260928 04:48:40.962296 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 04:48:40.962467 392 webserver.cc:460] Webserver started at http://127.0.0.1:34419/ using document root <none> and password file <none>
2026-09-28T04:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:40Z Disabled control of system clock
I20260928 04:48:40.969633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38961
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38961
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46009/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/testchainca.cert with env {}
W20260928 04:48:41.049715 5602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:41.049912 5602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:41.049933 5602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:41.051030 5602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:41.051070 5602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:41.051082 5602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:41.051126 5602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:41.051148 5602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:48:41.051174 5602 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:48:41.051220 5602 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46009/jwks.json
I20260928 04:48:41.052660 5602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46009/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:41.052902 5602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:41.053112 5602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:41.057631 5608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:41.057641 5607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:41.057758 5602 server_base.cc:1034] running on GCE node
W20260928 04:48:41.057632 5610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:41.058012 5602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:41.058290 5602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:41.059445 5602 hybrid_clock.cc:648] HybridClock initialized: now 1790570921059440 us; error 33 us; skew 500 ppm
I20260928 04:48:41.060525 5602 webserver.cc:460] Webserver started at http://127.0.98.62:42391/ using document root <none> and password file <none>
I20260928 04:48:41.060766 5602 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:41.060833 5602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:41.060977 5602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:41.062065 5602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca"
format_stamp: "Formatted at 2026-09-28 04:48:41 on dist-test-slave-jccs"
server_key: "2019b53903dff1aaab8a62ab4b55d998"
server_key_iv: "839f10da23ca454c213e3afab8e9f823"
server_key_version: "encryptionkey@0"
I20260928 04:48:41.062445 5602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca"
format_stamp: "Formatted at 2026-09-28 04:48:41 on dist-test-slave-jccs"
server_key: "2019b53903dff1aaab8a62ab4b55d998"
server_key_iv: "839f10da23ca454c213e3afab8e9f823"
server_key_version: "encryptionkey@0"
I20260928 04:48:41.063687 5602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:41.064342 5616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:41.064478 5602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:41.064613 5602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca"
format_stamp: "Formatted at 2026-09-28 04:48:41 on dist-test-slave-jccs"
server_key: "2019b53903dff1aaab8a62ab4b55d998"
server_key_iv: "839f10da23ca454c213e3afab8e9f823"
server_key_version: "encryptionkey@0"
I20260928 04:48:41.064699 5602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:41.087165 5602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:41.087479 5602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:41.087668 5602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:41.091588 5602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38961
I20260928 04:48:41.091636 5668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38961 every 8 connection(s)
I20260928 04:48:41.091987 5602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:41.092947 5669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:41.093286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 5602
I20260928 04:48:41.093392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:41.093575 392 external_mini_cluster.cc:1428] Setting key 0a339f1329f5db8081a04881617ff3b2
I20260928 04:48:41.096195 5669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca: Bootstrap starting.
I20260928 04:48:41.096843 5669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:41.097126 5669 log.cc:824] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:41.097829 5669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca: No bootstrap required, opened a new log
I20260928 04:48:41.099627 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:41.094093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.0.98.62:38961)
0928 04:48:41.094355 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:41.094364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:41.094396 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:48:41.095020 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:41.095028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:41.095103 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:41.095249 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:41.095454 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.096797 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.097627 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.098196 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.098318 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:41.098946 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:41.098951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:41.098951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:41.099151 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:41.099192 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:41.099205 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:41.099270 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:41.099334 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 04:48:41.099408 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260928 04:48:41.100363 5669 raft_consensus.cc:348] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } }
I20260928 04:48:41.100549 5669 raft_consensus.cc:374] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:41.100589 5669 raft_consensus.cc:729] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae4f075d0b64eb2a2a08d1a9b9c62ca, State: Initialized, Role: FOLLOWER
I20260928 04:48:41.100725 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:48:41.100713 5669 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } }
I20260928 04:48:41.100800 5669 raft_consensus.cc:388] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:41.100855 5669 raft_consensus.cc:482] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:41.100905 5669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:41.101570 5669 raft_consensus.cc:504] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } }
I20260928 04:48:41.101670 5669 leader_election.cc:302] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae4f075d0b64eb2a2a08d1a9b9c62ca; no voters:
I20260928 04:48:41.101815 5669 leader_election.cc:288] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:41.101872 5674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:41.102021 5674 raft_consensus.cc:686] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [term 1 LEADER]: Becoming Leader. State: Replica: aae4f075d0b64eb2a2a08d1a9b9c62ca, State: Running, Role: LEADER
I20260928 04:48:41.102113 5674 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } }
I20260928 04:48:41.102214 5669 sys_catalog.cc:564] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:41.102403 5674 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae4f075d0b64eb2a2a08d1a9b9c62ca. Latest consensus state: current_term: 1 leader_uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } } }
I20260928 04:48:41.102413 5675 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae4f075d0b64eb2a2a08d1a9b9c62ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } } }
I20260928 04:48:41.102480 5674 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:41.102504 5675 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:41.103436 5678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:41.103822 5678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:41.105242 5678 catalog_manager.cc:1347] Generated new cluster ID: 86af9154a1a54f0eb57605482bbfb748
I20260928 04:48:41.105293 5678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:41.122416 5678 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:41.122906 5678 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:41.130111 5678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aae4f075d0b64eb2a2a08d1a9b9c62ca: Generated new TSK 0
I20260928 04:48:41.130286 5678 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 04:48:41.149860 5699 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 04:48:41.149930 5699 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 04:48:41.150215 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:41.136602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.0.98.62:38961)
0928 04:48:41.136664 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:41.136666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:41.136768 (+ 102us) server_negotiation.cc:407] Connection header received
0928 04:48:41.136831 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:41.136832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:41.136841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:41.136895 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:48:41.137024 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.137524 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.138541 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.138710 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.138749 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:41.138910 (+ 161us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:48:41.150047 (+ 11137us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 04:48:41.150152 (+ 105us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 04:48:41.150278 5672 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 04:48:41.150375 5698 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:41.136553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38961 (local address 127.0.0.1:52280)
0928 04:48:41.136702 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:41.136712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:41.136772 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:41.136908 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:41.136909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:41.136911 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:48:41.136968 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:41.136970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.137591 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.137591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:41.138462 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:41.138465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.138766 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.138769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:41.138823 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:41.138828 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:48:41.150152 (+ 11324us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 04:48:41.150272 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38961: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46009/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260928 04:48:41.150646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 5602
2026-09-28T04:48:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 04:48:41.187595 392 mini_oidc.cc:152] Starting JWKS server
I20260928 04:48:41.188241 392 webserver.cc:460] Webserver started at https://127.0.0.1:35229/ using document root <none> and password file <none>
I20260928 04:48:41.188279 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 04:48:41.188411 392 webserver.cc:460] Webserver started at http://127.0.0.1:38635/ using document root <none> and password file <none>
2026-09-28T04:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:41Z Disabled control of system clock
I20260928 04:48:41.195542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37007
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37007
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35229/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 04:48:41.269935 5705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:41.270157 5705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:41.270188 5705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:41.271235 5705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:41.271294 5705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:41.271314 5705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:41.271334 5705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:41.271360 5705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:48:41.271381 5705 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:48:41.271407 5705 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35229/jwks.json
I20260928 04:48:41.272598 5705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35229/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:41.272853 5705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:41.273121 5705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:41.277807 5710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:41.277920 5705 server_base.cc:1034] running on GCE node
W20260928 04:48:41.277784 5711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:41.277799 5713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:41.278276 5705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:41.278579 5705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:41.279742 5705 hybrid_clock.cc:648] HybridClock initialized: now 1790570921279722 us; error 41 us; skew 500 ppm
I20260928 04:48:41.280961 5705 webserver.cc:460] Webserver started at http://127.0.98.62:38459/ using document root <none> and password file <none>
I20260928 04:48:41.281232 5705 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:41.281311 5705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:41.281440 5705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:41.282423 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "cc9730650d8c4a19ac37113949d7e738"
format_stamp: "Formatted at 2026-09-28 04:48:41 on dist-test-slave-jccs"
server_key: "1f747d05b4028402244902934c93412d"
server_key_iv: "a51a800449244a38a4c3ac2ca96fc962"
server_key_version: "encryptionkey@0"
I20260928 04:48:41.282804 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc9730650d8c4a19ac37113949d7e738"
format_stamp: "Formatted at 2026-09-28 04:48:41 on dist-test-slave-jccs"
server_key: "1f747d05b4028402244902934c93412d"
server_key_iv: "a51a800449244a38a4c3ac2ca96fc962"
server_key_version: "encryptionkey@0"
I20260928 04:48:41.284050 5705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:41.284791 5719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:41.284961 5705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:41.285068 5705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "cc9730650d8c4a19ac37113949d7e738"
format_stamp: "Formatted at 2026-09-28 04:48:41 on dist-test-slave-jccs"
server_key: "1f747d05b4028402244902934c93412d"
server_key_iv: "a51a800449244a38a4c3ac2ca96fc962"
server_key_version: "encryptionkey@0"
I20260928 04:48:41.285156 5705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:41.322677 5705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:41.323015 5705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:41.323172 5705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:41.327147 5705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37007
I20260928 04:48:41.327183 5771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37007 every 8 connection(s)
I20260928 04:48:41.327548 5705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:41.328184 5772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:41.329455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 5705
I20260928 04:48:41.329542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:41.329727 392 external_mini_cluster.cc:1428] Setting key 355e572f9e28ae280e6328b966b96b07
I20260928 04:48:41.331379 5772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738: Bootstrap starting.
I20260928 04:48:41.331924 5772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:41.332222 5772 log.cc:824] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:41.332891 5772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738: No bootstrap required, opened a new log
I20260928 04:48:41.334595 5772 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9730650d8c4a19ac37113949d7e738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37007 } }
I20260928 04:48:41.334735 5772 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:41.334796 5772 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9730650d8c4a19ac37113949d7e738, State: Initialized, Role: FOLLOWER
I20260928 04:48:41.334940 5772 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9730650d8c4a19ac37113949d7e738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37007 } }
I20260928 04:48:41.335031 5772 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:41.335067 5772 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:41.335119 5772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:41.335322 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:41.330202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.0.98.62:37007)
0928 04:48:41.330407 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:41.330416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:41.330447 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:48:41.331114 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:41.331124 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:41.331204 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:41.331351 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:41.331561 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.332511 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.333386 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.333819 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.333958 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:41.334599 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:41.334603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:41.334603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:41.334803 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:41.334843 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:41.334857 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:41.334918 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:41.335006 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 04:48:41.335100 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260928 04:48:41.335711 5772 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9730650d8c4a19ac37113949d7e738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37007 } }
I20260928 04:48:41.335834 5772 leader_election.cc:302] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc9730650d8c4a19ac37113949d7e738; no voters:
I20260928 04:48:41.335984 5772 leader_election.cc:288] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:41.336033 5777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:41.336181 5777 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [term 1 LEADER]: Becoming Leader. State: Replica: cc9730650d8c4a19ac37113949d7e738, State: Running, Role: LEADER
I20260928 04:48:41.336249 5772 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:41.336279 5777 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9730650d8c4a19ac37113949d7e738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37007 } }
I20260928 04:48:41.336326 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:48:41.336602 5777 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc9730650d8c4a19ac37113949d7e738. Latest consensus state: current_term: 1 leader_uuid: "cc9730650d8c4a19ac37113949d7e738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9730650d8c4a19ac37113949d7e738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37007 } } }
I20260928 04:48:41.336676 5777 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:41.336611 5779 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc9730650d8c4a19ac37113949d7e738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9730650d8c4a19ac37113949d7e738" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37007 } } }
I20260928 04:48:41.336823 5779 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738 [sys.catalog]: This master's current role is: LEADER
W20260928 04:48:41.338019 5792 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:48:41.338076 5792 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:48:41.338115 5785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:41.338248 5785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:41.339538 5785 catalog_manager.cc:1347] Generated new cluster ID: b8763ae4b6f94a95b738645cc1e5a06a
I20260928 04:48:41.339598 5785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:41.348786 5785 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:41.349313 5785 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:41.353964 5785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc9730650d8c4a19ac37113949d7e738: Generated new TSK 0
I20260928 04:48:41.354149 5785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:48:41.362746 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:41.359399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58068 (local address 127.0.98.62:37007)
0928 04:48:41.359442 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:41.359444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:41.359530 (+ 86us) server_negotiation.cc:407] Connection header received
0928 04:48:41.359611 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:41.359611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:41.359620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:41.359671 (+ 51us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:48:41.359785 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.360210 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.361227 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.361395 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.361454 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:41.361745 (+ 291us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:48:41.362548 (+ 803us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 04:48:41.362676 (+ 128us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58068: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 04:48:41.362797 5775 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58068: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 04:48:41.362982 5801 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:41.359314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37007 (local address 127.0.0.1:58068)
0928 04:48:41.359462 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:41.359472 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:41.359549 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:41.359679 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:41.359680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:41.359681 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:48:41.359732 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:41.359733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.360295 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.360296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:41.361116 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:41.361118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.361470 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.361476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:41.361530 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:41.361535 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:48:41.362652 (+ 1117us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 04:48:41.362749 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37007: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260928 04:48:41.363194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 5705
E20260928 04:48:41.373392 5802 webserver.cc:533] Webserver: sslize failed:
E20260928 04:48:41.373467 5802 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T04:48:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (212 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T04:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:41Z Disabled control of system clock
I20260928 04:48:41.407191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42373
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42373
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/testchainca.cert with env {}
W20260928 04:48:41.481220 5806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:41.481462 5806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:41.481498 5806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:41.482671 5806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:41.482743 5806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:41.482764 5806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:41.482784 5806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:41.482815 5806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 04:48:41.482837 5806 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 04:48:41.482860 5806 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 04:48:41.484339 5806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:41.484647 5806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:41.484859 5806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:41.489566 5812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:41.489566 5811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:41.489594 5814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:41.489805 5806 server_base.cc:1034] running on GCE node
I20260928 04:48:41.490036 5806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:41.490329 5806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:41.491470 5806 hybrid_clock.cc:648] HybridClock initialized: now 1790570921491460 us; error 27 us; skew 500 ppm
I20260928 04:48:41.492617 5806 webserver.cc:460] Webserver started at http://127.0.98.62:42897/ using document root <none> and password file <none>
I20260928 04:48:41.492858 5806 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:41.492901 5806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:41.493037 5806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:41.494097 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "5063611ec277457d90d80fe73995929b"
format_stamp: "Formatted at 2026-09-28 04:48:41 on dist-test-slave-jccs"
server_key: "dc41aec5a231138fe7175af04811a0f7"
server_key_iv: "71d0991bdc9fa004cba726e8e5e424e5"
server_key_version: "encryptionkey@0"
I20260928 04:48:41.494434 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "5063611ec277457d90d80fe73995929b"
format_stamp: "Formatted at 2026-09-28 04:48:41 on dist-test-slave-jccs"
server_key: "dc41aec5a231138fe7175af04811a0f7"
server_key_iv: "71d0991bdc9fa004cba726e8e5e424e5"
server_key_version: "encryptionkey@0"
I20260928 04:48:41.495604 5806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:41.496249 5820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:41.496431 5806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:41.496513 5806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "5063611ec277457d90d80fe73995929b"
format_stamp: "Formatted at 2026-09-28 04:48:41 on dist-test-slave-jccs"
server_key: "dc41aec5a231138fe7175af04811a0f7"
server_key_iv: "71d0991bdc9fa004cba726e8e5e424e5"
server_key_version: "encryptionkey@0"
I20260928 04:48:41.496577 5806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:41.512562 5806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:41.512843 5806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:41.512982 5806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:41.517165 5806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42373
I20260928 04:48:41.517243 5872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42373 every 8 connection(s)
I20260928 04:48:41.517568 5806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:41.518414 5873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:41.520534 5873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b: Bootstrap starting.
I20260928 04:48:41.520879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 5806
I20260928 04:48:41.520949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:41.521116 5873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:41.521123 392 external_mini_cluster.cc:1428] Setting key f66b84ef881b39a5cd3d70da623b8add
I20260928 04:48:41.521449 5873 log.cc:824] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:41.522177 5873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b: No bootstrap required, opened a new log
I20260928 04:48:41.524708 5873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5063611ec277457d90d80fe73995929b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } }
I20260928 04:48:41.524832 5873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:41.524854 5873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5063611ec277457d90d80fe73995929b, State: Initialized, Role: FOLLOWER
I20260928 04:48:41.524987 5873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5063611ec277457d90d80fe73995929b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } }
I20260928 04:48:41.525046 5873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:41.525098 5873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:41.525192 5873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:41.525918 5873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5063611ec277457d90d80fe73995929b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } }
I20260928 04:48:41.526031 5873 leader_election.cc:302] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5063611ec277457d90d80fe73995929b; no voters:
I20260928 04:48:41.526183 5873 leader_election.cc:288] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:41.526288 5878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:41.526429 5873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:41.526444 5878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [term 1 LEADER]: Becoming Leader. State: Replica: 5063611ec277457d90d80fe73995929b, State: Running, Role: LEADER
I20260928 04:48:41.526852 5878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5063611ec277457d90d80fe73995929b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } }
I20260928 04:48:41.527208 5880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5063611ec277457d90d80fe73995929b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5063611ec277457d90d80fe73995929b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } } }
I20260928 04:48:41.527303 5880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:41.527580 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:41.521650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.0.98.62:42373)
0928 04:48:41.521800 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:41.521807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:41.521936 (+ 129us) server_negotiation.cc:407] Connection header received
0928 04:48:41.522554 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:41.522562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:41.522636 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:41.522783 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:41.522974 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.524422 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.525437 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.526063 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.526194 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:41.526889 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:41.526892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:41.526893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:48:41.527090 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:41.527132 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:48:41.527147 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:41.527207 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:41.527308 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 04:48:41.527384 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260928 04:48:41.527665 5879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5063611ec277457d90d80fe73995929b. Latest consensus state: current_term: 1 leader_uuid: "5063611ec277457d90d80fe73995929b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5063611ec277457d90d80fe73995929b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42373 } } }
I20260928 04:48:41.527743 5879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:41.528208 5886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:41.528694 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:48:41.528918 5886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:41.530341 5886 catalog_manager.cc:1347] Generated new cluster ID: 1a925ce466574774a50f27bffd5deffa
I20260928 04:48:41.530424 5886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:41.536929 5886 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:41.537444 5886 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:41.548205 5886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5063611ec277457d90d80fe73995929b: Generated new TSK 0
I20260928 04:48:41.548400 5886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:48:41.609756 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:41.553388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.0.98.62:42373)
0928 04:48:41.553449 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:41.553451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:41.553558 (+ 107us) server_negotiation.cc:407] Connection header received
0928 04:48:41.553623 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:41.553623 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:41.553647 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:41.553711 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0928 04:48:41.553852 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.554312 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.555412 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.555612 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.555667 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:41.555838 (+ 171us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 04:48:41.609559 (+ 53721us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 04:48:41.609698 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47214: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 04:48:41.609838 5877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47214: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 04:48:41.609969 5902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:41.553337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42373 (local address 127.0.0.1:47214)
0928 04:48:41.553486 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:41.553496 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:41.553560 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:41.553726 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:41.553727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:41.553729 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 04:48:41.553794 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:41.553796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.554381 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.554382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:41.555336 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:41.555338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:41.555682 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:41.555687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:41.555740 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:41.555745 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 04:48:41.609707 (+ 53962us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 04:48:41.609858 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42373: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260928 04:48:41.610313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 5806
2026-09-28T04:48:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (226 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:41 dist-test-slave-jccs krb5kdc[5914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:41 dist-test-slave-jccs krb5kdc[5914](info): set up 2 sockets
Sep 28 04:48:41 dist-test-slave-jccs krb5kdc[5914](info): commencing operation
krb5kdc: starting...
W20260928 04:48:42.801326 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:42 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:42.912006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45909
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:42.988054 5927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:42.988279 5927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:42.988315 5927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:42.989467 5927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:42.989532 5927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:42.989557 5927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:42.989583 5927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:42.989612 5927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:42.991042 5927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:42.991291 5927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:42.991500 5927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:48:42.996229 5927 server_base.cc:1034] running on GCE node
W20260928 04:48:42.996197 5933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:42.996202 5932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:42.996220 5935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:42.996614 5927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:42.996829 5927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:42.998018 5927 hybrid_clock.cc:648] HybridClock initialized: now 1790570922998011 us; error 35 us; skew 500 ppm
Sep 28 04:48:42 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:43.003834 5927 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:43.004235 5927 webserver.cc:460] Webserver started at http://127.0.98.62:43449/ using document root <none> and password file <none>
I20260928 04:48:43.004453 5927 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:43.004518 5927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:43.004660 5927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:43.005750 5927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "371cc9c72f0942d9b63fbb579f558d23"
format_stamp: "Formatted at 2026-09-28 04:48:42 on dist-test-slave-jccs"
server_key: "565ccf308dc1014eadd872f2c41a3e47"
server_key_iv: "f2e90aa0b8e693e23ff3bb9cbeaa8b68"
server_key_version: "encryptionkey@0"
I20260928 04:48:43.006104 5927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "371cc9c72f0942d9b63fbb579f558d23"
format_stamp: "Formatted at 2026-09-28 04:48:42 on dist-test-slave-jccs"
server_key: "565ccf308dc1014eadd872f2c41a3e47"
server_key_iv: "f2e90aa0b8e693e23ff3bb9cbeaa8b68"
server_key_version: "encryptionkey@0"
I20260928 04:48:43.007295 5927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:43.007937 5942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:43.008100 5927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:43.008193 5927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "371cc9c72f0942d9b63fbb579f558d23"
format_stamp: "Formatted at 2026-09-28 04:48:42 on dist-test-slave-jccs"
server_key: "565ccf308dc1014eadd872f2c41a3e47"
server_key_iv: "f2e90aa0b8e693e23ff3bb9cbeaa8b68"
server_key_version: "encryptionkey@0"
I20260928 04:48:43.008266 5927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:43.024875 5927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:43.027489 5927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:43.027627 5927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:43.032590 5927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45909
I20260928 04:48:43.033042 5927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:43.033273 5994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45909 every 8 connection(s)
I20260928 04:48:43.033708 5995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:43.035992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 5927
I20260928 04:48:43.036108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:43.036331 392 external_mini_cluster.cc:1428] Setting key 7c76e51aa7eb2b6487f258d8ee30146d
I20260928 04:48:43.037123 5995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23: Bootstrap starting.
I20260928 04:48:43.037886 5995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:43.038225 5995 log.cc:824] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:43.038941 5995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23: No bootstrap required, opened a new log
I20260928 04:48:43.040724 5995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371cc9c72f0942d9b63fbb579f558d23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } }
I20260928 04:48:43.040845 5995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:43.040867 5995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371cc9c72f0942d9b63fbb579f558d23, State: Initialized, Role: FOLLOWER
I20260928 04:48:43.040992 5995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371cc9c72f0942d9b63fbb579f558d23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } }
I20260928 04:48:43.041085 5995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:43.041122 5995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:43.041195 5995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:43.041787 5995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371cc9c72f0942d9b63fbb579f558d23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } }
I20260928 04:48:43.041910 5995 leader_election.cc:302] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 371cc9c72f0942d9b63fbb579f558d23; no voters:
I20260928 04:48:43.042055 5995 leader_election.cc:288] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:43.042089 6000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:43.042214 6000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [term 1 LEADER]: Becoming Leader. State: Replica: 371cc9c72f0942d9b63fbb579f558d23, State: Running, Role: LEADER
I20260928 04:48:43.042289 5995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:43.042296 6000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371cc9c72f0942d9b63fbb579f558d23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } }
I20260928 04:48:43.042838 6001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "371cc9c72f0942d9b63fbb579f558d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371cc9c72f0942d9b63fbb579f558d23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } }
I20260928 04:48:43.042924 6001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:43.043078 6002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 371cc9c72f0942d9b63fbb579f558d23. Latest consensus state: current_term: 1 leader_uuid: "371cc9c72f0942d9b63fbb579f558d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371cc9c72f0942d9b63fbb579f558d23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45909 } } }
I20260928 04:48:43.043139 6002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23 [sys.catalog]: This master's current role is: LEADER
Sep 28 04:48:43 dist-test-slave-jccs krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:43.043246 6007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:43.043905 6007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:43.046186 6007 catalog_manager.cc:1347] Generated new cluster ID: 83484b41e4d4432793b18eb69735e53e
I20260928 04:48:43.046243 6007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:43.047148 5998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:43.036825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.0.98.62:45909)
0928 04:48:43.037033 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:43.037042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:43.037072 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:48:43.037691 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:43.037700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:43.037761 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:43.037940 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:43.038680 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:43.039800 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:43.040515 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:43.040984 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:43.041091 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:43.043711 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:43.043716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:43.043716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:43.043727 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:43.045527 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:43.045954 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:43.045957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:43.045957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:43.046058 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:43.046264 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:43.046265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:43.046266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:43.046526 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:43.046615 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:43.046830 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 04:48:43.046955 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
I20260928 04:48:43.056813 6007 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:43.057330 6007 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:43.063606 6007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 371cc9c72f0942d9b63fbb579f558d23: Generated new TSK 0
I20260928 04:48:43.063763 6007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260928 04:48:43.645438 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.575s user 0.001s sys 0.002s
I20260928 04:48:43.645774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45909
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:43.721935 6021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:43.722124 6021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:43.722146 6021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:43.723179 6021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:43.723212 6021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:43.723235 6021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:43.724562 6021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:43.724718 6021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:43.724877 6021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:43.729480 6026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:43.729573 6029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:43.729494 6027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:43.729526 6021 server_base.cc:1034] running on GCE node
I20260928 04:48:43.729868 6021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:43.730084 6021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:43.731289 6021 hybrid_clock.cc:648] HybridClock initialized: now 1790570923731275 us; error 25 us; skew 500 ppm
Sep 28 04:48:43 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:43.733780 6021 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:48:43.734225 6021 webserver.cc:460] Webserver started at http://127.0.98.1:36079/ using document root <none> and password file <none>
I20260928 04:48:43.734436 6021 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:43.734499 6021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:43.734647 6021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:43.735598 6021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "84e1b53048114d33be42f1a50df6f9f4"
format_stamp: "Formatted at 2026-09-28 04:48:43 on dist-test-slave-jccs"
server_key: "32685871e950423696c4e916a0531992"
server_key_iv: "8aca170cde63b950e2589ef927713600"
server_key_version: "encryptionkey@0"
I20260928 04:48:43.735934 6021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "84e1b53048114d33be42f1a50df6f9f4"
format_stamp: "Formatted at 2026-09-28 04:48:43 on dist-test-slave-jccs"
server_key: "32685871e950423696c4e916a0531992"
server_key_iv: "8aca170cde63b950e2589ef927713600"
server_key_version: "encryptionkey@0"
I20260928 04:48:43.737159 6021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:48:43.737952 6036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:43.738114 6021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:43.738204 6021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "84e1b53048114d33be42f1a50df6f9f4"
format_stamp: "Formatted at 2026-09-28 04:48:43 on dist-test-slave-jccs"
server_key: "32685871e950423696c4e916a0531992"
server_key_iv: "8aca170cde63b950e2589ef927713600"
server_key_version: "encryptionkey@0"
I20260928 04:48:43.738283 6021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:43.757272 6021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:43.759388 6021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:43.759529 6021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:43.759737 6021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:43.760023 6021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:43.760074 6021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:43.760113 6021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:43.760138 6021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:43.766319 6021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45851
I20260928 04:48:43.766359 6149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45851 every 8 connection(s)
I20260928 04:48:43.766665 6021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:48:43.770110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 6021
I20260928 04:48:43.770238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:43.770416 392 external_mini_cluster.cc:1428] Setting key 1842725bc37a681cbceec33c8a7933b8
Sep 28 04:48:43 dist-test-slave-jccs krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:43.776448 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:43.767458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34127 (local address 127.0.98.62:45909)
0928 04:48:43.767594 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:43.767597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:43.767737 (+ 140us) server_negotiation.cc:407] Connection header received
0928 04:48:43.768701 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:43.768703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:43.768716 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:43.768789 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:43.769447 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:43.769976 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:43.771201 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:43.771513 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:43.771586 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:43.773767 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:43.773768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:43.773769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:43.773777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:43.775214 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:43.775599 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:43.775600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:43.775600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:43.775657 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:43.775876 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:43.775876 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:43.775877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:43.775982 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:43.776048 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:43.776288 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 04:48:43.776365 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260928 04:48:43.776613 6152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:43.767427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45909 (local address 127.0.98.1:34127)
0928 04:48:43.767604 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:43.767647 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:43.768590 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:43.768855 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:43.768859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:43.769108 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:43.769358 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:43.769362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:43.770064 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:43.770065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:43.771126 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:43.771129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:43.771593 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:43.771594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:43.771691 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:43.772178 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:43.772193 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:43.773681 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:43.775308 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:43.775314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:43.775316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:43.775510 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:43.775717 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:43.775717 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:43.775718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:43.775812 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:43.776048 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:43.776054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:43.776174 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 04:48:43.776269 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:48:43.776400 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260928 04:48:43.777115 6150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45909
I20260928 04:48:43.777211 6150 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:43.777380 6150 heartbeater.cc:511] Master 127.0.98.62:45909 requested a full tablet report, sending...
I20260928 04:48:43.777967 5959 ts_manager.cc:194] Registered new tserver with Master: 84e1b53048114d33be42f1a50df6f9f4 (127.0.98.1:45851)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:48:43.778749 5959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:48:43.795334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45909
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:43.871657 6156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:43.871825 6156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:43.871847 6156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:43.872851 6156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:43.872884 6156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:43.872928 6156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:43.874329 6156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:43.874493 6156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:43.874634 6156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:43.879122 6162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:43.879074 6161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:43.879166 6156 server_base.cc:1034] running on GCE node
W20260928 04:48:43.879093 6164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:43.879557 6156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:43.879781 6156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:43.880915 6156 hybrid_clock.cc:648] HybridClock initialized: now 1790570923880905 us; error 27 us; skew 500 ppm
Sep 28 04:48:43 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:43.883316 6156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:48:43.883698 6156 webserver.cc:460] Webserver started at http://127.0.98.2:39487/ using document root <none> and password file <none>
I20260928 04:48:43.883909 6156 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:43.883951 6156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:43.884084 6156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:43.885073 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "0fd7d144012b44c9bfa44a0392434ba0"
format_stamp: "Formatted at 2026-09-28 04:48:43 on dist-test-slave-jccs"
server_key: "f4a6b0073b55bc0708762316703e0df9"
server_key_iv: "97249332c03111a8128908cb65f7f1b6"
server_key_version: "encryptionkey@0"
I20260928 04:48:43.885470 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0fd7d144012b44c9bfa44a0392434ba0"
format_stamp: "Formatted at 2026-09-28 04:48:43 on dist-test-slave-jccs"
server_key: "f4a6b0073b55bc0708762316703e0df9"
server_key_iv: "97249332c03111a8128908cb65f7f1b6"
server_key_version: "encryptionkey@0"
I20260928 04:48:43.886708 6156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:43.887344 6171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:43.887498 6156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:43.887599 6156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "0fd7d144012b44c9bfa44a0392434ba0"
format_stamp: "Formatted at 2026-09-28 04:48:43 on dist-test-slave-jccs"
server_key: "f4a6b0073b55bc0708762316703e0df9"
server_key_iv: "97249332c03111a8128908cb65f7f1b6"
server_key_version: "encryptionkey@0"
I20260928 04:48:43.887668 6156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:43.896308 6156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:43.898524 6156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:43.898680 6156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:43.898876 6156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:43.899142 6156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:43.899192 6156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:43.899230 6156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:43.899253 6156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:43.904420 6156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37125
I20260928 04:48:43.904480 6284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37125 every 8 connection(s)
I20260928 04:48:43.904764 6156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:48:43.909065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 6156
I20260928 04:48:43.909147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:43.909351 392 external_mini_cluster.cc:1428] Setting key de8c9a2d117f962d225c093c5a1427d3
Sep 28 04:48:43 dist-test-slave-jccs krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:43.914105 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:43.905480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56237 (local address 127.0.98.62:45909)
0928 04:48:43.905539 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:43.905541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:43.905720 (+ 179us) server_negotiation.cc:407] Connection header received
0928 04:48:43.906626 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:43.906628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:43.906642 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:43.906705 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:43.907270 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:43.907751 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:43.908714 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:43.908929 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:43.909012 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:43.911255 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:43.911257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:43.911257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:43.911265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:43.912721 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:43.913136 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:43.913137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:43.913137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:43.913238 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:43.913487 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:43.913488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:43.913489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:43.913602 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:43.913674 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:43.913935 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 04:48:43.914015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:48:43.914290 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:43.905456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45909 (local address 127.0.98.2:56237)
0928 04:48:43.905605 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:43.905642 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:43.906516 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:43.906770 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:43.906774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:43.907002 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:43.907176 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:43.907180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:43.907839 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:43.907840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:43.908625 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:43.908627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:43.909019 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:43.909020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:43.909166 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:43.909737 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:43.909744 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:43.911157 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:43.912818 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:43.912824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:43.912826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:43.913060 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:43.913309 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:43.913310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:43.913311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:43.913418 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:43.913678 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:43.913684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:43.913812 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 04:48:43.913914 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 04:48:43.914069 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260928 04:48:43.914736 6285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45909
I20260928 04:48:43.914819 6285 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:43.914995 6285 heartbeater.cc:511] Master 127.0.98.62:45909 requested a full tablet report, sending...
I20260928 04:48:43.915345 5959 ts_manager.cc:194] Registered new tserver with Master: 0fd7d144012b44c9bfa44a0392434ba0 (127.0.98.2:37125)
I20260928 04:48:43.915697 5959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56237
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:48:43.933828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45909
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:44.010690 6290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:44.010856 6290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:44.010890 6290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:44.012029 6290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:44.012072 6290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:44.012127 6290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:44.013638 6290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:44.013859 6290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:44.014070 6290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:44.018627 6296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:44.018666 6298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:44.018682 6295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:44.018736 6290 server_base.cc:1034] running on GCE node
I20260928 04:48:44.019052 6290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:44.019351 6290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:44.020504 6290 hybrid_clock.cc:648] HybridClock initialized: now 1790570924020497 us; error 37 us; skew 500 ppm
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:44.022977 6290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:48:44.023413 6290 webserver.cc:460] Webserver started at http://127.0.98.3:37589/ using document root <none> and password file <none>
I20260928 04:48:44.023643 6290 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:44.023711 6290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:44.023934 6290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:44.025000 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5e05703958349bcb4634afbdfe4f723"
format_stamp: "Formatted at 2026-09-28 04:48:44 on dist-test-slave-jccs"
server_key: "8cd9005abda3dbb516367f26f94b9da0"
server_key_iv: "2403635b467b6724a04ebcfb02a0931b"
server_key_version: "encryptionkey@0"
I20260928 04:48:44.025383 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5e05703958349bcb4634afbdfe4f723"
format_stamp: "Formatted at 2026-09-28 04:48:44 on dist-test-slave-jccs"
server_key: "8cd9005abda3dbb516367f26f94b9da0"
server_key_iv: "2403635b467b6724a04ebcfb02a0931b"
server_key_version: "encryptionkey@0"
I20260928 04:48:44.026670 6290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:44.027331 6305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:44.027542 6290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:44.027634 6290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "a5e05703958349bcb4634afbdfe4f723"
format_stamp: "Formatted at 2026-09-28 04:48:44 on dist-test-slave-jccs"
server_key: "8cd9005abda3dbb516367f26f94b9da0"
server_key_iv: "2403635b467b6724a04ebcfb02a0931b"
server_key_version: "encryptionkey@0"
I20260928 04:48:44.027714 6290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:44.060570 6290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:44.062938 6290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:44.063117 6290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:44.063366 6290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:44.063696 6290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:44.063748 6290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:44.063778 6290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:44.063800 6290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:44.069600 6290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35273
I20260928 04:48:44.069633 6418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35273 every 8 connection(s)
I20260928 04:48:44.069940 6290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:44.077927 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:44.070589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58383 (local address 127.0.98.62:45909)
0928 04:48:44.070650 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:44.070652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:44.070859 (+ 207us) server_negotiation.cc:407] Connection header received
0928 04:48:44.071710 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:44.071711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:44.071719 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:44.071768 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:44.072266 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:44.072680 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:44.073712 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:44.073883 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:44.073936 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:44.075915 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:44.075916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:44.075916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:44.075922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:44.076959 (+ 1037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:44.077316 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:44.077316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:44.077317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:44.077361 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:44.077535 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:44.077535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:44.077535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:44.077611 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:44.077649 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:44.077819 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 04:48:44.077872 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:48:44.078119 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:44.070575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45909 (local address 127.0.98.3:58383)
0928 04:48:44.070733 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:44.070781 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:44.071631 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:44.071816 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:44.071820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:44.072034 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:44.072207 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:44.072211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:44.072744 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:44.072745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:44.073639 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:44.073641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:44.073971 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:44.073972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:44.074065 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:44.074570 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:44.074577 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:44.075830 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:44.077022 (+ 1192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:44.077026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:44.077028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:44.077262 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:44.077417 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:44.077417 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:44.077418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:44.077501 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:44.077653 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:44.077657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:44.077744 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 04:48:44.077810 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 04:48:44.077904 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260928 04:48:44.078219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 6290
I20260928 04:48:44.078302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:44.078480 392 external_mini_cluster.cc:1428] Setting key a6f32a709789f19f3c1c550cd361b78a
I20260928 04:48:44.078612 6419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45909
I20260928 04:48:44.078708 6419 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:44.078898 6419 heartbeater.cc:511] Master 127.0.98.62:45909 requested a full tablet report, sending...
I20260928 04:48:44.079285 5956 ts_manager.cc:194] Registered new tserver with Master: a5e05703958349bcb4634afbdfe4f723 (127.0.98.3:35273)
I20260928 04:48:44.079620 5956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58383
I20260928 04:48:44.089993 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/test-admin.keytab.
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:44.105903 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 04:48:44.107309 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:44.106639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.0.98.1:45851)
0928 04:48:44.106805 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:44.106810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:44.106819 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:44.106842 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:44.106844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:44.106856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:44.106919 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:44.107216 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37084: BlockingRecv error: recv got EOF from 127.0.0.1:37084 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
W20260928 04:48:44.107426 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5cc)
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:44.108322 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:44.109110 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:48:44.116278 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:44.109637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.0.98.1:45851)
0928 04:48:44.109690 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:44.109692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:44.109699 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:48:44.109734 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:44.109735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:44.109743 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:44.109786 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:44.110189 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:44.110925 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:44.111643 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:44.111980 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:44.112032 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:44.113968 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:44.113974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:44.113974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:44.113980 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:44.115433 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:44.115725 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:44.115728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:44.115728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:44.115775 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:44.115918 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:44.115918 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:44.115919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:44.116025 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:44.116131 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:44.116165 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 04:48:44.116217 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 04:48:44.117893 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:44.117424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.0.98.1:45851)
0928 04:48:44.117458 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:44.117460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:44.117489 (+ 29us) server_negotiation.cc:407] Connection header received
0928 04:48:44.117543 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:44.117544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:44.117552 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:44.117592 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:44.117853 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37102: BlockingRecv error: recv got EOF from 127.0.0.1:37102 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 04:48:44.117938 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:44.118768 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:44.119562 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:48:44.125243 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:44.119948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.0.98.1:45851)
0928 04:48:44.119997 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:44.119999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:44.120005 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:48:44.120060 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:44.120061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:44.120067 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:44.120104 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:44.120383 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:44.120779 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:44.121383 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:44.121551 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:44.121598 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:44.123271 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:44.123271 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:44.123272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:44.123276 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:44.124359 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:44.124647 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:44.124647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:44.124647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:44.124690 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:44.124822 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:44.124823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:44.124823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:44.124887 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:44.124973 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:44.125054 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 04:48:44.125106 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 04:48:44.126693 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:44.126172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.0.98.1:45851)
0928 04:48:44.126202 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:44.126203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:44.126255 (+ 52us) server_negotiation.cc:407] Connection header received
0928 04:48:44.126325 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:44.126325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:44.126331 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:44.126375 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:44.126628 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37128: BlockingRecv error: recv got EOF from 127.0.0.1:37128 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 04:48:44.126799 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790570897511889-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:44.127653 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:44.128445 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[5914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:48:44.134743 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:44.128876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.0.98.1:45851)
0928 04:48:44.128927 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:44.128929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:44.128974 (+ 45us) server_negotiation.cc:407] Connection header received
0928 04:48:44.129033 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:44.129034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:44.129042 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:44.129109 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:44.129497 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:44.129936 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:44.130585 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:44.130736 (+ 151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:44.130778 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:44.132607 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:44.132608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:44.132609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:44.132614 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:44.133994 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:44.134269 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:44.134269 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:44.134270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:44.134312 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:44.134448 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:44.134448 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:44.134448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:44.134514 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:44.134569 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:44.134648 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 04:48:44.134695 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:48:44.135267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 6021
I20260928 04:48:44.140522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 6156
I20260928 04:48:44.144620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 6290
I20260928 04:48:44.149250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 5927
2026-09-28T04:48:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2369 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[6439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[6439](info): set up 2 sockets
Sep 28 04:48:44 dist-test-slave-jccs krb5kdc[6439](info): commencing operation
krb5kdc: starting...
W20260928 04:48:45.137382 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 04:48:45.247205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43173
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:45.321728 6452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:45.321951 6452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:45.321986 6452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:45.323156 6452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:45.323215 6452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:45.323242 6452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:45.323272 6452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:45.323302 6452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:45.324702 6452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:45.324949 6452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:45.325225 6452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:45.329687 6457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:45.329751 6460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:45.329713 6458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:45.329978 6452 server_base.cc:1034] running on GCE node
I20260928 04:48:45.330204 6452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:45.330405 6452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:45.331558 6452 hybrid_clock.cc:648] HybridClock initialized: now 1790570925331552 us; error 42 us; skew 500 ppm
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:45.333827 6452 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 04:48:45.334205 6452 webserver.cc:460] Webserver started at http://127.0.98.62:41537/ using document root <none> and password file <none>
I20260928 04:48:45.334432 6452 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:45.334496 6452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:45.334636 6452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:45.335556 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "c1d7afa74ea042a0bfcd16202957fa7d"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "e7a8571a28f141fdf8a5dd8304f07842"
server_key_iv: "db6c4a8f94e0cdb10c4a2dffbb8b8dc2"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.335868 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1d7afa74ea042a0bfcd16202957fa7d"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "e7a8571a28f141fdf8a5dd8304f07842"
server_key_iv: "db6c4a8f94e0cdb10c4a2dffbb8b8dc2"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.337092 6452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:45.337798 6467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.337952 6452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:45.338043 6452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "c1d7afa74ea042a0bfcd16202957fa7d"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "e7a8571a28f141fdf8a5dd8304f07842"
server_key_iv: "db6c4a8f94e0cdb10c4a2dffbb8b8dc2"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.338120 6452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:45.367272 6452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:45.369431 6452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:45.369565 6452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:45.372969 6452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43173
I20260928 04:48:45.373001 6519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43173 every 8 connection(s)
I20260928 04:48:45.373417 6452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:45.374292 6520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:45.376215 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d: Bootstrap starting.
I20260928 04:48:45.376703 6520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:45.376951 6520 log.cc:824] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:45.377597 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d: No bootstrap required, opened a new log
I20260928 04:48:45.379030 6520 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } }
I20260928 04:48:45.379144 6520 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:45.379176 6520 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d7afa74ea042a0bfcd16202957fa7d, State: Initialized, Role: FOLLOWER
I20260928 04:48:45.379259 6520 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } }
I20260928 04:48:45.379364 6520 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:45.379390 6520 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:45.379431 6520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:45.380110 6520 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } }
I20260928 04:48:45.380215 6520 leader_election.cc:302] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1d7afa74ea042a0bfcd16202957fa7d; no voters:
I20260928 04:48:45.380370 6520 leader_election.cc:288] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:45.380424 6523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:45.380568 6523 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 1 LEADER]: Becoming Leader. State: Replica: c1d7afa74ea042a0bfcd16202957fa7d, State: Running, Role: LEADER
I20260928 04:48:45.380591 6520 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:45.380645 6523 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } }
I20260928 04:48:45.380967 6524 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } }
I20260928 04:48:45.380973 6525 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1d7afa74ea042a0bfcd16202957fa7d. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } }
I20260928 04:48:45.381064 6525 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:45.381281 6524 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:45.381400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 6452
I20260928 04:48:45.381513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:45.381573 6531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:45.381688 392 external_mini_cluster.cc:1428] Setting key cd827d3002db6bd7d28ff7a92eda5268
I20260928 04:48:45.382015 6531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:45.383509 6531 catalog_manager.cc:1347] Generated new cluster ID: eac889087153461c93f3e51a5e012818
I20260928 04:48:45.383579 6531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:45.388259 6531 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:45.388917 6531 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:45.394205 6531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d: Generated new TSK 0
I20260928 04:48:45.394354 6531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:48:45.395409 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.382262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48452 (local address 127.0.98.62:43173)
0928 04:48:45.382409 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:45.382417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:45.382796 (+ 379us) server_negotiation.cc:407] Connection header received
0928 04:48:45.382907 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:45.382912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:45.382968 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:45.383095 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:45.388096 (+ 5001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.388998 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.389717 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.390094 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.390212 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.392259 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:45.392262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:45.392262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:45.392279 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:45.393849 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.394332 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.394336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.394337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.394486 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.394706 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.394707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.394707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.394901 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:45.395041 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:45.395126 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 04:48:45.395256 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 04:48:45.418455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43173
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:45.494010 6546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:45.494228 6546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:45.494268 6546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:45.495388 6546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:45.495450 6546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:45.495492 6546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:45.496855 6546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:45.497105 6546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:45.497341 6546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:45.502005 6552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:45.501998 6554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:45.502069 6551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:45.502144 6546 server_base.cc:1034] running on GCE node
I20260928 04:48:45.502336 6546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:45.502548 6546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:45.503705 6546 hybrid_clock.cc:648] HybridClock initialized: now 1790570925503700 us; error 34 us; skew 500 ppm
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:45.506201 6546 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 04:48:45.506628 6546 webserver.cc:460] Webserver started at http://127.0.98.1:36543/ using document root <none> and password file <none>
I20260928 04:48:45.506796 6546 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:45.506831 6546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:45.506909 6546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:45.507776 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "14aef32c5a874dc39da827c6377d5e59"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "f7e278f6e587d011e2335c97bf2c1b30"
server_key_iv: "9bce8eea59778fa7b2334ed5c10ab652"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.508107 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "14aef32c5a874dc39da827c6377d5e59"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "f7e278f6e587d011e2335c97bf2c1b30"
server_key_iv: "9bce8eea59778fa7b2334ed5c10ab652"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.509344 6546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:45.509965 6561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.510109 6546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.510186 6546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "14aef32c5a874dc39da827c6377d5e59"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "f7e278f6e587d011e2335c97bf2c1b30"
server_key_iv: "9bce8eea59778fa7b2334ed5c10ab652"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.510236 6546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:45.519546 6546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:45.521162 6546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:45.521319 6546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:45.521474 6546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:45.521709 6546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:45.521750 6546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.521768 6546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:45.521790 6546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.527325 6546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40905
I20260928 04:48:45.527400 6674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40905 every 8 connection(s)
I20260928 04:48:45.527666 6546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:48:45.532301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 6546
I20260928 04:48:45.532377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:45.532541 392 external_mini_cluster.cc:1428] Setting key ddc852dccfadfa3bc81976bd9506311a
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:45.536518 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.528309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.1:42255)
0928 04:48:45.528450 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:45.528486 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:45.528907 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:45.529109 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:45.529112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:45.529704 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:45.529874 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:45.529878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.530498 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.530499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:45.531247 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:45.531248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.531568 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.531569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:45.531660 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.532068 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:45.532074 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:45.533591 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:45.535197 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:45.535202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:45.535204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:45.535500 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:45.535729 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:45.535730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:45.535730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:45.535840 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:45.536095 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:45.536099 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:45.536195 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 04:48:45.536266 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:48:45.536367 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260928 04:48:45.536544 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.528347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42255 (local address 127.0.98.62:43173)
0928 04:48:45.528410 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:45.528412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:45.528562 (+ 150us) server_negotiation.cc:407] Connection header received
0928 04:48:45.528981 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:45.528981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:45.528990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:45.529043 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:45.529949 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.530425 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.531322 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.531503 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.531551 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.533684 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:45.533685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:45.533686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:45.533693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:45.535118 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.535584 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.535585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.535585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.535653 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.535902 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.535903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.535903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.536036 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:45.536099 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:45.536309 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 04:48:45.536396 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:48:45.537047 6675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43173
I20260928 04:48:45.537132 6675 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:45.537341 6675 heartbeater.cc:511] Master 127.0.98.62:43173 requested a full tablet report, sending...
I20260928 04:48:45.537918 6484 ts_manager.cc:194] Registered new tserver with Master: 14aef32c5a874dc39da827c6377d5e59 (127.0.98.1:40905)
I20260928 04:48:45.538827 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42255
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 04:48:45.557523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43173
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:45.635008 6680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:45.635185 6680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:45.635213 6680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:45.636216 6680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:45.636245 6680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:45.636269 6680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:45.637550 6680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:45.637712 6680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:45.637859 6680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:45.642582 6685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:45.642635 6688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:45.642635 6686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:45.642877 6680 server_base.cc:1034] running on GCE node
I20260928 04:48:45.643041 6680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:45.643216 6680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:45.644363 6680 hybrid_clock.cc:648] HybridClock initialized: now 1790570925644348 us; error 30 us; skew 500 ppm
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:45.646749 6680 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 04:48:45.647126 6680 webserver.cc:460] Webserver started at http://127.0.98.2:45119/ using document root <none> and password file <none>
I20260928 04:48:45.647334 6680 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:45.647375 6680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:45.647517 6680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:45.648501 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "138d57a007d94250a5297be6e3b6fd04"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "ad6148c40682669873b028dd0e0f652c"
server_key_iv: "f3e81938e815dc15ccbf421db8628573"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.648836 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "138d57a007d94250a5297be6e3b6fd04"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "ad6148c40682669873b028dd0e0f652c"
server_key_iv: "f3e81938e815dc15ccbf421db8628573"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.650120 6680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:48:45.650745 6695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.650907 6680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.651005 6680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "138d57a007d94250a5297be6e3b6fd04"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "ad6148c40682669873b028dd0e0f652c"
server_key_iv: "f3e81938e815dc15ccbf421db8628573"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.651077 6680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:45.669538 6680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:45.671799 6680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:45.671900 6680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:45.672052 6680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:45.672273 6680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:45.672307 6680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.672327 6680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:45.672340 6680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.677726 6680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37101
I20260928 04:48:45.677764 6808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37101 every 8 connection(s)
I20260928 04:48:45.678037 6680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:48:45.681692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 6680
I20260928 04:48:45.681788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:45.682029 392 external_mini_cluster.cc:1428] Setting key 874b62ee2ca84cb2599a02f724254f06
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:48:45.691511 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.678697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47475 (local address 127.0.98.62:43173)
0928 04:48:45.678754 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:45.678756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:45.678953 (+ 197us) server_negotiation.cc:407] Connection header received
0928 04:48:45.679374 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:45.679374 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:45.679383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:45.679447 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:45.680247 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.680776 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.681968 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.682274 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.682340 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.684728 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:45.684729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:45.684729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:45.684736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:45.690352 (+ 5616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.690743 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.690744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.690745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.690794 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.690990 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.690991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.690991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.691086 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:45.691194 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:45.691380 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 04:48:45.691454 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:48:45.691642 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.678682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.2:47475)
0928 04:48:45.678833 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:45.678869 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:45.679298 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:45.679494 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:45.679496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:45.680001 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:45.680172 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:45.680176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.680841 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.680841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:45.681822 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:45.681825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.682355 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.682357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:45.682468 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.682950 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:45.682960 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:45.684632 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:45.690478 (+ 5846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:45.690483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:45.690485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:45.690680 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:45.690851 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:45.690852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:45.690852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:45.690939 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:45.691182 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:45.691185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:45.691285 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 04:48:45.691364 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 04:48:45.691478 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260928 04:48:45.692122 6809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43173
I20260928 04:48:45.692245 6809 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:45.692425 6809 heartbeater.cc:511] Master 127.0.98.62:43173 requested a full tablet report, sending...
I20260928 04:48:45.692819 6484 ts_manager.cc:194] Registered new tserver with Master: 138d57a007d94250a5297be6e3b6fd04 (127.0.98.2:37101)
I20260928 04:48:45.693256 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 04:48:45.706183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43173
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:45.781987 6814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:45.782176 6814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:45.782215 6814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:45.783347 6814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:45.783411 6814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:45.783456 6814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:45.784830 6814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:45.785063 6814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:45.785329 6814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:45.789830 6822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:45.789842 6820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:45.789860 6819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:45.790143 6814 server_base.cc:1034] running on GCE node
I20260928 04:48:45.790335 6814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:45.790503 6814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:45.791627 6814 hybrid_clock.cc:648] HybridClock initialized: now 1790570925791603 us; error 39 us; skew 500 ppm
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:45.793846 6814 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 04:48:45.794183 6814 webserver.cc:460] Webserver started at http://127.0.98.3:38121/ using document root <none> and password file <none>
I20260928 04:48:45.794346 6814 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:45.794384 6814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:45.794464 6814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:45.795297 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "79efdd818fc44a249647d6c2409d9d4d"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "8db35077349bbaacaf64bf9d63275e80"
server_key_iv: "ea4ba53289359b3391571ba7f8f9eca0"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.795558 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "79efdd818fc44a249647d6c2409d9d4d"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "8db35077349bbaacaf64bf9d63275e80"
server_key_iv: "ea4ba53289359b3391571ba7f8f9eca0"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.796643 6814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:48:45.797248 6829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.797413 6814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.797498 6814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "79efdd818fc44a249647d6c2409d9d4d"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "8db35077349bbaacaf64bf9d63275e80"
server_key_iv: "ea4ba53289359b3391571ba7f8f9eca0"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.797577 6814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:45.822571 6814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:45.824314 6814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:45.824483 6814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:45.824707 6814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:45.825011 6814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:45.825060 6814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.825100 6814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:45.825142 6814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.830633 6814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42081
I20260928 04:48:45.830678 6942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42081 every 8 connection(s)
I20260928 04:48:45.830956 6814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:45.838821 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.831550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44601 (local address 127.0.98.62:43173)
0928 04:48:45.831598 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:45.831601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:45.831837 (+ 236us) server_negotiation.cc:407] Connection header received
0928 04:48:45.832247 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:45.832248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:45.832257 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:45.832306 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:45.833096 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.833646 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.834571 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.834733 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.834778 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.836496 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:45.836498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:45.836498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:45.836503 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:45.837831 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.838124 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.838125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.838125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.838169 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.838368 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.838369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.838369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.838457 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:45.838503 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:45.838704 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 04:48:45.838762 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:48:45.839000 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.831547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.3:44601)
0928 04:48:45.831726 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:45.831758 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:45.832171 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:45.832373 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:45.832376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:45.832868 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:45.833029 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:45.833033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.833717 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.833717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:45.834489 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:45.834490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.834792 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.834792 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:45.834882 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.835254 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:45.835261 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:45.836421 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:45.837887 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:45.837891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:45.837893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:45.838066 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:45.838218 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:45.838218 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:45.838218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:45.838317 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:45.838505 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:45.838508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:45.838604 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 04:48:45.838676 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:48:45.838777 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260928 04:48:45.839443 6943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43173
I20260928 04:48:45.839514 6943 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:45.839684 6943 heartbeater.cc:511] Master 127.0.98.62:43173 requested a full tablet report, sending...
I20260928 04:48:45.840062 6484 ts_manager.cc:194] Registered new tserver with Master: 79efdd818fc44a249647d6c2409d9d4d (127.0.98.3:42081)
I20260928 04:48:45.840422 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44601
I20260928 04:48:45.840771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 6814
I20260928 04:48:45.840873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:45.841090 392 external_mini_cluster.cc:1428] Setting key a7997a5d1eb19086854e95b7490d74aa
I20260928 04:48:45.841918 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260928 04:48:45.865746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40337
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43173,127.0.98.61:40337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:45.942595 6948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:45.942777 6948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:45.942797 6948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:45.943856 6948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:45.943889 6948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:45.943903 6948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:45.943915 6948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:45.943928 6948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:45.945327 6948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43173,127.0.98.61:40337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
W20260928 04:48:45.945416 6948 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43173,127.0.98.61:40337'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 04:48:45.945515 6948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:45.945670 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:45.950078 6953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:45.950073 6954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:45.950073 6956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:45.950271 6948 server_base.cc:1034] running on GCE node
I20260928 04:48:45.950516 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:45.950788 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:45.951970 6948 hybrid_clock.cc:648] HybridClock initialized: now 1790570925951955 us; error 32 us; skew 500 ppm
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:45.954767 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 04:48:45.955228 6948 webserver.cc:460] Webserver started at http://127.0.98.61:37183/ using document root <none> and password file <none>
I20260928 04:48:45.955456 6948 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:45.955518 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:45.955667 6948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:45.956598 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data/instance:
uuid: "33ed8f11d6494de087aa6ae314418ea5"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "2bcf785fa43940c98400ac59ca8d940d"
server_key_iv: "44b2c3913f439cc45a96928cb0ff8fd1"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.956941 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal/instance:
uuid: "33ed8f11d6494de087aa6ae314418ea5"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "2bcf785fa43940c98400ac59ca8d940d"
server_key_iv: "44b2c3913f439cc45a96928cb0ff8fd1"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.958250 6948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:48:45.958889 6963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:45.959036 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:45.959148 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal
uuid: "33ed8f11d6494de087aa6ae314418ea5"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "2bcf785fa43940c98400ac59ca8d940d"
server_key_iv: "44b2c3913f439cc45a96928cb0ff8fd1"
server_key_version: "encryptionkey@0"
I20260928 04:48:45.959231 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:45.976682 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:45.978518 6948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:45.978675 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:45.982563 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40337
I20260928 04:48:45.982596 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40337 every 8 connection(s)
I20260928 04:48:45.983017 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
I20260928 04:48:45.983958 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:45.985550 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:48:45.990003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 6948
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:45.994376 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.986220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.0.98.62:43173)
0928 04:48:45.986264 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:45.986266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:45.986488 (+ 222us) server_negotiation.cc:407] Connection header received
0928 04:48:45.986961 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:45.986963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:45.986976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:45.987030 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:45.987613 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.988085 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.989063 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.989303 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.989383 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.991449 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:45.991450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:45.991451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:45.991459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:45.992936 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.993362 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.993363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.993363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.993422 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.993634 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.993634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.993635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.993841 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:45.993911 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:45.994243 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 04:48:45.994315 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:48:45.994663 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.986219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:48462)
0928 04:48:45.986377 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:45.986408 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:45.986852 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:45.987110 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:45.987114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:45.987346 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:45.987515 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:45.987519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.988182 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.988183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:45.988975 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:45.988977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.989390 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.989391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:45.989511 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.989912 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:45.989922 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:45.991363 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:45.993014 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:45.993018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:45.993020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:45.993290 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:45.993484 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:45.993485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:45.993485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:45.993573 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:45.993914 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:45.993919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:45.994076 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 04:48:45.994197 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 04:48:45.994352 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260928 04:48:45.995296 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 04:48:45.998790 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.990457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.0.98.61:40337)
0928 04:48:45.990604 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:45.990609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:45.990632 (+ 23us) server_negotiation.cc:407] Connection header received
0928 04:48:45.990653 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:45.990655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:45.990673 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:45.990767 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:45.991358 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.992192 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.993264 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.993720 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.993786 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.996182 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:45.996187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:45.996187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:45.996193 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:45.997612 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.997948 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.997950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.997951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.998006 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:45.998215 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:45.998216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:45.998216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:45.998371 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:45.998515 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:45.998629 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 04:48:45.998697 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
Sep 28 04:48:45 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 04:48:46.002806 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.995528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40337 (local address 127.0.0.1:60350)
0928 04:48:45.995572 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:45.995581 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:45.995616 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:45.996141 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:45.996142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:45.996268 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:45.996307 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:45.996308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.997243 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.997244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:45.997731 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:45.997733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.998118 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.998119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:45.998190 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:45.998580 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:45.998584 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:45.999955 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.002135 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.002136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.002136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.002276 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.002420 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.002420 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.002420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.002463 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.002617 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.002618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.002645 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.002692 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.002754 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 04:48:46.002919 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:45.995881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.0.98.61:40337)
0928 04:48:45.996009 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:45.996011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:45.996018 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:48:45.996037 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:45.996038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:45.996049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:45.996105 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:45.996371 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.997110 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.997807 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:45.998054 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:45.998109 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.000053 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.000054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.000054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.000059 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.002071 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.002330 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.002331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.002331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.002370 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.002499 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.002499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.002499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.002572 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.002613 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.002704 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.002795 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
I20260928 04:48:46.004010 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Bootstrap starting.
I20260928 04:48:46.004575 7016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:46.004834 7016 log.cc:824] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:46.005569 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: No bootstrap required, opened a new log
I20260928 04:48:46.006843 7016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } }
I20260928 04:48:46.006953 7016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:46.006996 7016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ed8f11d6494de087aa6ae314418ea5, State: Initialized, Role: FOLLOWER
I20260928 04:48:46.007114 7016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } }
I20260928 04:48:46.007334 7023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } } }
I20260928 04:48:46.007378 7016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:46.007431 7023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:48:46.008910 7034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:48:46.008963 7034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:48:46.010351 6948 master_runner.cc:428] Detected that this master 33ed8f11d6494de087aa6ae314418ea5 is joining an existing cluster
I20260928 04:48:46.010406 6948 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:40337
I20260928 04:48:46.018072 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.011462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40337 (local address 127.0.0.1:60356)
0928 04:48:46.011574 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.011584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.011638 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.011787 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.011788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.012040 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.012135 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.012136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.013006 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.013007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.013717 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.013719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.014198 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.014199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.014265 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.014673 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.014678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.015062 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.016930 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.016930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.016931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.017146 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.017397 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.017398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.017398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.017468 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.017705 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.017706 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.017758 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.017831 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.017976 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":13,"thread_start_us":62,"threads_started":1}
I20260928 04:48:46.018436 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.011513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.0.98.61:40337)
0928 04:48:46.011542 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.011544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.011643 (+ 99us) server_negotiation.cc:407] Connection header received
0928 04:48:46.011710 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.011710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.011717 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.011740 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.012209 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.012930 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.013803 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.014105 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.014171 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.015144 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.015145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.015145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.015151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.016839 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.017268 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.017269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.017269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.017326 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.017529 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.017530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.017530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.017629 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.017666 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.017871 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.018079 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":16}
I20260928 04:48:46.019004 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.011323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.0.98.62:43173)
0928 04:48:46.011360 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.011362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.011659 (+ 297us) server_negotiation.cc:407] Connection header received
0928 04:48:46.011703 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.011704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.011713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.011787 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.013028 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.013798 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.014713 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.014945 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.015029 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.016029 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.016030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.016030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.016036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.017625 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.018170 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.018171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.018171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.018231 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.018420 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.018421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.018422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.018550 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.018617 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.018830 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.018908 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:48:46.019260 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.011250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:48470)
0928 04:48:46.011574 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.011584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.011652 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.011791 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.011791 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.012045 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.012135 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.012136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.013876 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.013876 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.014615 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.014617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.015031 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.015032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.015092 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.015471 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.015475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.015920 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.017735 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.017735 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.017736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.018075 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.018296 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.018297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.018297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.018348 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.018653 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.018654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.018690 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.019080 (+ 390us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.019164 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 04:48:46.027549 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.021109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.0.98.61:40337)
0928 04:48:46.021145 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.021147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.021291 (+ 144us) server_negotiation.cc:407] Connection header received
0928 04:48:46.021391 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.021392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.021401 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.021443 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.021831 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.022244 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.022996 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.023181 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.023260 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.025051 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.025052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.025052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.025058 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.026405 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.026778 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.026779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.026780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.026832 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.027052 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.027053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.027053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.027144 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.027213 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.027376 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.027447 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:48:46.027596 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.021057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40337 (local address 127.0.0.1:60372)
0928 04:48:46.021163 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.021224 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.021314 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.021458 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.021463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.021707 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.021745 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.021746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.022320 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.022321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.022914 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.022915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.023260 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.023261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.023346 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.023767 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.023771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.024967 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.026498 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.026505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.026506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.026694 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.026903 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.026904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.026904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.026960 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.027208 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.027215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.027261 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.027360 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.027461 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":60,"threads_started":1}
I20260928 04:48:46.028029 6484 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:40337: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 40337
}
attrs {
promote: true
}
}
dest_uuid: "c1d7afa74ea042a0bfcd16202957fa7d"
cas_config_opid_index: -1
I20260928 04:48:46.028340 6484 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } }
I20260928 04:48:46.028820 7044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 33ed8f11d6494de087aa6ae314418ea5 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } } }
I20260928 04:48:46.029042 6990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:46.029789 6990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Refusing update from remote peer c1d7afa74ea042a0bfcd16202957fa7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 04:48:46.029830 7044 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:40337
I20260928 04:48:46.029966 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } } }
I20260928 04:48:46.030014 6948 master.cc:537] Master@127.0.98.61:40337 shutting down...
I20260928 04:48:46.030030 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:46.030119 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 04:48:46.030925 6468 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 04:48:46.031575 6468 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d -> Peer 33ed8f11d6494de087aa6ae314418ea5 (127.0.98.61:40337): Couldn't send request to peer 33ed8f11d6494de087aa6ae314418ea5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 04:48:46.032739 6948 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:46.032802 6948 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:46.032871 6948 tablet_replica.cc:330] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: stopping tablet replica
I20260928 04:48:46.045037 6948 master.cc:552] Master@127.0.98.61:40337 shutdown complete.
I20260928 04:48:46.045071 6948 master_runner.cc:305] Clearing existing system tablet
I20260928 04:48:46.045243 6948 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:46.045280 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:46.045877 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:46.046334 7048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.046474 6948 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:46.046535 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.046578 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal
uuid: "33ed8f11d6494de087aa6ae314418ea5"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "2bcf785fa43940c98400ac59ca8d940d"
server_key_iv: "44b2c3913f439cc45a96928cb0ff8fd1"
server_key_version: "encryptionkey@0"
I20260928 04:48:46.047122 6948 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:46.048468 6948 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 04:48:46.048521 6948 log.cc:1196] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 04:48:46.048702 6948 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Deleting consensus metadata
I20260928 04:48:46.048821 6948 master_runner.cc:315] Copying system tablet from 127.0.98.62:43173
I20260928 04:48:46.049407 6948 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43173
I20260928 04:48:46.055459 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.049719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:48478)
0928 04:48:46.049818 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.049828 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.049880 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.050018 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.050018 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.050153 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.050202 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.050204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.051101 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.051101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.051658 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.051659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.052116 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.052117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.052164 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.052620 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.052624 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.052896 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.054603 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.054604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.054605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.054755 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.054945 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.054945 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.054945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.054991 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.055234 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.055234 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.055265 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.055314 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.055388 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":59,"threads_started":1}
I20260928 04:48:46.055495 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.049828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.0.98.62:43173)
0928 04:48:46.049869 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.049871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.049880 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:46.049929 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.049930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.049940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.050016 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.050264 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.051025 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.051733 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.052042 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.052114 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.052999 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.053001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.053001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.053008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.054530 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.054824 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.054825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.054825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.054884 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.055048 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.055049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.055049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.055174 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.055240 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.055336 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.055405 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:48:46.056093 6504 tablet_copy_service.cc:140] P c1d7afa74ea042a0bfcd16202957fa7d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 33ed8f11d6494de087aa6ae314418ea5 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48478)
I20260928 04:48:46.056185 6504 tablet_copy_service.cc:161] P c1d7afa74ea042a0bfcd16202957fa7d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 33ed8f11d6494de087aa6ae314418ea5 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48478: session id = 33ed8f11d6494de087aa6ae314418ea5-00000000000000000000000000000000
I20260928 04:48:46.056751 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 04:48:46.057318 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 04:48:46.058104 6948 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: tablet copy: Starting download of 0 data blocks...
I20260928 04:48:46.058255 6948 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: tablet copy: Starting download of 1 WAL segments...
I20260928 04:48:46.059048 6948 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 04:48:46.060496 6504 tablet_copy_service.cc:342] P c1d7afa74ea042a0bfcd16202957fa7d: Request end of tablet copy session 33ed8f11d6494de087aa6ae314418ea5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48478
I20260928 04:48:46.060563 6504 tablet_copy_service.cc:433] P c1d7afa74ea042a0bfcd16202957fa7d: ending tablet copy session 33ed8f11d6494de087aa6ae314418ea5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 33ed8f11d6494de087aa6ae314418ea5
I20260928 04:48:46.066272 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:46.067515 7056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:46.067528 7055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:46.074057 7058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:46.074213 6948 server_base.cc:1034] running on GCE node
I20260928 04:48:46.074365 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:46.074502 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:46.075680 6948 hybrid_clock.cc:648] HybridClock initialized: now 1790570926075671 us; error 33 us; skew 500 ppm
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:46.077020 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 04:48:46.077287 6948 webserver.cc:460] Webserver started at http://127.0.98.61:43219/ using document root <none> and password file <none>
I20260928 04:48:46.077395 6948 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:46.077435 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:46.078078 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:46.078580 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.078724 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.078768 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal
uuid: "33ed8f11d6494de087aa6ae314418ea5"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "2bcf785fa43940c98400ac59ca8d940d"
server_key_iv: "44b2c3913f439cc45a96928cb0ff8fd1"
server_key_version: "encryptionkey@0"
I20260928 04:48:46.078820 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:46.088023 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:46.089946 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:46.093113 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40337
I20260928 04:48:46.093597 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
I20260928 04:48:46.094439 7118 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:48:46.094929 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Bootstrap starting.
I20260928 04:48:46.097609 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:48:46.097831 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Bootstrap complete.
I20260928 04:48:46.098003 7118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } }
I20260928 04:48:46.098081 7118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 33ed8f11d6494de087aa6ae314418ea5, State: Initialized, Role: LEARNER
I20260928 04:48:46.098114 7118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } }
I20260928 04:48:46.098282 7118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:46.098270 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } } }
I20260928 04:48:46.098377 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:48:46.099689 7132 catalog_manager.cc:1259] Loaded cluster ID: eac889087153461c93f3e51a5e012818
I20260928 04:48:46.099777 7132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: loading cluster ID for follower catalog manager: success
I20260928 04:48:46.101460 7132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: acquiring CA information for follower catalog manager: success
I20260928 04:48:46.101799 7132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:48:46.105311 7117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40337 every 8 connection(s)
I20260928 04:48:46.113294 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.105462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.98.61:40337)
0928 04:48:46.105561 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.105563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.105573 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:46.105604 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.105605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.105614 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.105683 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.106399 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.107385 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.108550 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.108922 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.109001 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.110544 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.110545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.110545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.110552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.112081 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.112406 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.112407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.112407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.112469 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.112750 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.112751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.112752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.112882 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.112995 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.113055 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.113128 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":54,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260928 04:48:46.136068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37139
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43173,127.0.98.61:40337,127.0.98.60:37139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:46.215233 7136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:46.215461 7136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:46.215494 7136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:46.216523 7136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:46.216580 7136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:46.216621 7136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:46.216674 7136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:46.216702 7136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:46.218129 7136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43173,127.0.98.61:40337,127.0.98.60:37139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:46.218385 7136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:46.218602 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:46.222879 7141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:46.223014 7136 server_base.cc:1034] running on GCE node
W20260928 04:48:46.223109 7142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:46.223165 7144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:46.223477 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:46.223703 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:46.224861 7136 hybrid_clock.cc:648] HybridClock initialized: now 1790570926224847 us; error 28 us; skew 500 ppm
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:46.227219 7136 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 04:48:46.227656 7136 webserver.cc:460] Webserver started at http://127.0.98.60:40477/ using document root <none> and password file <none>
I20260928 04:48:46.227861 7136 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:46.227924 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:46.228067 7136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:46.229051 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data/instance:
uuid: "ce55491dfcb148e7b3515700c9c0620a"
format_stamp: "Formatted at 2026-09-28 04:48:46 on dist-test-slave-jccs"
server_key: "1699d94d31fcf5429c8933cba1a99398"
server_key_iv: "4a7bdd848eebf1a3cf6f0debcba6a426"
server_key_version: "encryptionkey@0"
I20260928 04:48:46.229422 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal/instance:
uuid: "ce55491dfcb148e7b3515700c9c0620a"
format_stamp: "Formatted at 2026-09-28 04:48:46 on dist-test-slave-jccs"
server_key: "1699d94d31fcf5429c8933cba1a99398"
server_key_iv: "4a7bdd848eebf1a3cf6f0debcba6a426"
server_key_version: "encryptionkey@0"
I20260928 04:48:46.230621 7136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:46.231216 7151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.231361 7136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:46.231454 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal
uuid: "ce55491dfcb148e7b3515700c9c0620a"
format_stamp: "Formatted at 2026-09-28 04:48:46 on dist-test-slave-jccs"
server_key: "1699d94d31fcf5429c8933cba1a99398"
server_key_iv: "4a7bdd848eebf1a3cf6f0debcba6a426"
server_key_version: "encryptionkey@0"
I20260928 04:48:46.231520 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:46.252247 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:46.254647 7136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:46.254814 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:46.259290 7136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37139
I20260928 04:48:46.259316 7203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37139 every 8 connection(s)
I20260928 04:48:46.259644 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
I20260928 04:48:46.260372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 7136
I20260928 04:48:46.260627 7204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:46.262879 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:46.270496 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.260959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.0.98.60:37139)
0928 04:48:46.261099 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.261106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.261127 (+ 21us) server_negotiation.cc:407] Connection header received
0928 04:48:46.261667 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.261673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.261715 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.261815 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.262531 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.263722 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.264643 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.265156 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.265315 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.267656 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.267663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.267663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.267669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.269062 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.269420 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.269423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.269425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.269493 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.269683 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.269684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.269685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.269909 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.270041 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.270176 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.270298 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260928 04:48:46.270663 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.263590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.0.98.62:43173)
0928 04:48:46.263643 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.263646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.263663 (+ 17us) server_negotiation.cc:407] Connection header received
0928 04:48:46.263750 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.263751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.263763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.263815 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.264337 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.264929 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.265916 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.266086 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.266131 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.268249 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.268250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.268250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.268257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.269593 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.269900 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.269901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.269901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.269956 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.270131 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.270132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.270132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.270245 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.270307 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.270499 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.270565 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:48:46.270809 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.263430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:48490)
0928 04:48:46.263581 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.263619 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.263691 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.263848 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.263850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.264178 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.264249 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.264251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.265012 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.265013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.265829 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.265831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.266162 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.266164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.266241 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.266675 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.266683 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.268176 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.269654 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.269658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.269660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.269828 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.270013 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.270013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.270014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.270069 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.270332 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.270336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.270384 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.270459 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.270654 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260928 04:48:46.271430 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 04:48:46.277518 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.271705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.98.61:40337)
0928 04:48:46.271762 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.271764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.271772 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:46.271802 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.271803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.271812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.271883 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.272538 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.272972 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.273748 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.273972 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.274055 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.275658 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.275659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.275659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.275666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.276707 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.276960 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.276960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.276961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.277001 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.277126 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.277127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.277127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.277260 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.277311 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.277404 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.277462 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:48:46.277544 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.271632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40337 (local address 127.0.0.1:60392)
0928 04:48:46.271673 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.271682 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.271715 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.271887 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.271888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.272029 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.272457 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.272459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.273059 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.273059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.273664 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.273666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.274056 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.274056 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.274126 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.274516 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.274521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.275574 (+ 1053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.276767 (+ 1193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.276768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.276768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.276903 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.277051 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.277051 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.277052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.277093 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.277313 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.277313 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.277342 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.277390 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.277448 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 04:48:46.277935 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 04:48:46.283546 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.278236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.0.98.60:37139)
0928 04:48:46.278285 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.278287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.278294 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:48:46.278311 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.278312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.278319 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.278375 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.278539 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.279052 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.279575 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.279790 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.279837 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.281692 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.281694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.281694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.281699 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.282768 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.283048 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.283048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.283049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.283091 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.283220 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.283220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.283220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.283296 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.283338 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.283433 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.283489 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:48:46.283660 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.278153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.0.1:60998)
0928 04:48:46.278195 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.278206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.278240 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.278379 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.278380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.278477 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.278500 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.278501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.279105 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.279105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.279512 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.279513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.279850 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.279850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.279883 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.280280 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.280285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.281629 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.282833 (+ 1204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.282834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.282835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.282989 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.283141 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.283142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.283142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.283185 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.283330 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.283330 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.283358 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.283419 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.283527 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":44}
I20260928 04:48:46.284896 7204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: Bootstrap starting.
I20260928 04:48:46.285713 7204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:46.285934 7204 log.cc:824] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:46.286522 7204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: No bootstrap required, opened a new log
I20260928 04:48:46.287767 7204 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } }
I20260928 04:48:46.287884 7204 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:46.287906 7204 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce55491dfcb148e7b3515700c9c0620a, State: Initialized, Role: FOLLOWER
I20260928 04:48:46.288019 7204 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } }
I20260928 04:48:46.288261 7209 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } } }
I20260928 04:48:46.288333 7204 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:46.288339 7209 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:48:46.290019 7220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:48:46.290076 7220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:48:46.292049 7136 master_runner.cc:428] Detected that this master ce55491dfcb148e7b3515700c9c0620a is joining an existing cluster
I20260928 04:48:46.292084 7136 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:37139
I20260928 04:48:46.299193 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.292963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:48502)
0928 04:48:46.293087 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.293098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.293154 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.293660 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.293661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.293910 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.293979 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.293980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.294627 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.294628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.295176 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.295178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.295571 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.295572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.295638 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.296087 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.296091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.296406 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.298178 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.298179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.298180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.298380 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.298599 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.298600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.298600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.298655 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.298898 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.298899 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.298940 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.299007 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.299089 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260928 04:48:46.299371 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.293038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.0.98.62:43173)
0928 04:48:46.293083 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.293085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.293142 (+ 57us) server_negotiation.cc:407] Connection header received
0928 04:48:46.293540 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.293542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.293554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.293605 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.294065 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.294541 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.295286 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.295483 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.295560 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.296534 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.296535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.296535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.296542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.298063 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.298456 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.298457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.298457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.298509 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.298720 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.298721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.298721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.298843 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.298903 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.299031 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.299105 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 04:48:46.309882 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.293167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40337 (local address 127.0.0.1:60402)
0928 04:48:46.293286 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.293294 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.293327 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.293684 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.293685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.293995 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.294054 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.294056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.294981 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.294982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.295831 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.295833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.296307 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.296308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.296408 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.297061 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.297066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.297534 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.299742 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.299743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.299743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.299974 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.300169 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.300169 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.300170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.309325 (+ 9155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.309606 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.309606 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.309653 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.309701 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.309785 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260928 04:48:46.310209 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.293488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.0.98.61:40337)
0928 04:48:46.293549 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.293550 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.293558 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:46.293577 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.293578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.293587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.293637 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.294149 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.294889 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.295924 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.296200 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.296238 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.298096 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.298098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.298098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.298104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.299623 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.300063 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.300064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.300065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.300116 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.309404 (+ 9288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.309405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.309405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.309534 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.309585 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.310035 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.310104 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:48:46.312444 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.301035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.0.1:32780)
0928 04:48:46.301076 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.301085 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.301121 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.301400 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.301401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.301554 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.301594 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.301596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.302165 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.302165 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.302913 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.302915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.303224 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.303225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.303284 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.303628 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.303633 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.304829 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.311427 (+ 6598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.311428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.311428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.311625 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.311836 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.311837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.311838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.311888 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.312143 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.312144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.312202 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.312267 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.312357 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 04:48:46.312475 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.301082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.0.98.60:37139)
0928 04:48:46.301243 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.301245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.301253 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:46.301303 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.301307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.301318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.301381 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.301670 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.302097 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.302995 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.303165 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.303204 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.304901 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.304902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.304903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.304909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.311362 (+ 6453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.311717 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.311718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.311719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.311775 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.311961 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.311962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.311962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.312073 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.312136 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.312291 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.312363 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260928 04:48:46.313297 6484 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:37139: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 37139
}
attrs {
promote: true
}
}
dest_uuid: "c1d7afa74ea042a0bfcd16202957fa7d"
cas_config_opid_index: 5
I20260928 04:48:46.313555 6484 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } }
I20260928 04:48:46.313949 7043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ce55491dfcb148e7b3515700c9c0620a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.314246 7178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:46.315037 7043 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:37139
I20260928 04:48:46.315115 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.315194 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:46.315387 7136 master.cc:537] Master@127.0.98.60:37139 shutting down...
I20260928 04:48:46.315687 7178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 FOLLOWER]: Refusing update from remote peer c1d7afa74ea042a0bfcd16202957fa7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 04:48:46.315936 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 04:48:46.316541 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d -> Peer ce55491dfcb148e7b3515700c9c0620a (127.0.98.60:37139): Couldn't send request to peer ce55491dfcb148e7b3515700c9c0620a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 04:48:46.316705 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.293393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.0.98.60:37139)
0928 04:48:46.293424 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.293426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.293533 (+ 107us) server_negotiation.cc:407] Connection header received
0928 04:48:46.293616 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.293617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.293625 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.293652 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.309505 (+ 15853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.310122 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.310963 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.311122 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.311194 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.312010 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.312011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.312012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.312017 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.313549 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.315965 (+ 2416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.315965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.315966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.316017 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.316197 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.316198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.316198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.316320 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.316361 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.316535 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.316604 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:48:46.316991 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.293360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.0.1:32778)
0928 04:48:46.293478 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.293488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.293554 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.293702 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.293702 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.309383 (+ 15681us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.309433 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.309434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.310195 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.310195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.310877 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.310879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.311195 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.311196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.311247 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.311614 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.311618 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.311938 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.313628 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.313629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.313629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.315858 (+ 2229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.316084 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.316084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.316085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.316134 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.316401 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.316401 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.316435 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.316788 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.316899 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":80,"threads_started":1}
I20260928 04:48:46.318517 7136 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:46.318600 7136 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:46.318648 7136 tablet_replica.cc:330] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: stopping tablet replica
I20260928 04:48:46.319816 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.314120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.0.98.61:40337)
0928 04:48:46.314244 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.314246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.314421 (+ 175us) server_negotiation.cc:407] Connection header received
0928 04:48:46.314442 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.314443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.314465 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.314518 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:46.314664 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.315405 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.318856 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.319403 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.319465 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.319542 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.319725 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.319756 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 04:48:46.319819 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.313987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40337 (local address 127.0.0.1:60408)
0928 04:48:46.314273 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.314283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.314339 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.314567 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.314567 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.314569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:46.314600 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.314602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.316906 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.316908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.318761 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.318764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.319576 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.319577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.319647 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.319669 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.319721 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.319756 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260928 04:48:46.320261 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 LEARNER]: Refusing update from remote peer c1d7afa74ea042a0bfcd16202957fa7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 04:48:46.320461 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 04:48:46.321030 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ce55491dfcb148e7b3515700c9c0620a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.322083 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.322243 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:48:46.322250 7044 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d: attempting to promote NON_VOTER 33ed8f11d6494de087aa6ae314418ea5 to VOTER
I20260928 04:48:46.322420 7044 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } }
I20260928 04:48:46.322561 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1d7afa74ea042a0bfcd16202957fa7d. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: true } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.322712 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: This master's current role is: LEARNER
I20260928 04:48:46.323016 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.322910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.0.1:32804)
0928 04:48:46.322947 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.322997 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 04:48:46.323005 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 LEARNER]: Refusing update from remote peer c1d7afa74ea042a0bfcd16202957fa7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 04:48:46.323217 7232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 04:48:46.323498 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.323417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.0.1:32818)
0928 04:48:46.323446 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.323470 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 04:48:46.323602 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d -> Peer ce55491dfcb148e7b3515700c9c0620a (127.0.98.60:37139): Couldn't send request to peer ce55491dfcb148e7b3515700c9c0620a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37139: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 04:48:46.324074 7043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 33ed8f11d6494de087aa6ae314418ea5 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.324776 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.324855 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.324732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.0.1:32834)
0928 04:48:46.324799 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.324831 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 04:48:46.324855 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:46.324913 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.324962 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:46.325168 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 33ed8f11d6494de087aa6ae314418ea5 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.326043 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.326124 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:48:46.331475 7136 master.cc:552] Master@127.0.98.60:37139 shutdown complete.
I20260928 04:48:46.331518 7136 master_runner.cc:305] Clearing existing system tablet
I20260928 04:48:46.331652 7136 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:46.331688 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:46.332239 7136 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:46.332669 7234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.332801 7136 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:46.332854 7136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.332875 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal
uuid: "ce55491dfcb148e7b3515700c9c0620a"
format_stamp: "Formatted at 2026-09-28 04:48:46 on dist-test-slave-jccs"
server_key: "1699d94d31fcf5429c8933cba1a99398"
server_key_iv: "4a7bdd848eebf1a3cf6f0debcba6a426"
server_key_version: "encryptionkey@0"
I20260928 04:48:46.333542 7136 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:46.334658 7136 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 04:48:46.334712 7136 log.cc:1196] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 04:48:46.334908 7136 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: Deleting consensus metadata
I20260928 04:48:46.335026 7136 master_runner.cc:315] Copying system tablet from 127.0.98.62:43173
I20260928 04:48:46.335628 7136 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43173
I20260928 04:48:46.341699 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.335958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.0.98.62:43173)
0928 04:48:46.336000 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.336002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.336066 (+ 64us) server_negotiation.cc:407] Connection header received
0928 04:48:46.336130 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.336130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.336137 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.336182 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.336526 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.336959 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.337989 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.338155 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.338204 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.339113 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.339115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.339115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.339121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.340585 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.340998 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.340998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.340999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.341052 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.341275 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.341275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.341290 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.341401 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.341451 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.341596 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.341647 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:48:46.341997 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.335880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:48506)
0928 04:48:46.336003 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.336014 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.336075 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.336198 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.336199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.336393 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.336461 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.336463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.337026 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.337027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.337918 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.337920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.338220 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.338221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.338283 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.338680 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.338685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.339036 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.340657 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.340659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.340659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.340880 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.341116 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.341117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.341117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.341205 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.341462 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.341463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.341504 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.341581 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.341672 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260928 04:48:46.342224 6504 tablet_copy_service.cc:140] P c1d7afa74ea042a0bfcd16202957fa7d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ce55491dfcb148e7b3515700c9c0620a ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48506)
I20260928 04:48:46.342278 6504 tablet_copy_service.cc:161] P c1d7afa74ea042a0bfcd16202957fa7d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ce55491dfcb148e7b3515700c9c0620a at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48506: session id = ce55491dfcb148e7b3515700c9c0620a-00000000000000000000000000000000
I20260928 04:48:46.342778 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 04:48:46.343225 7136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 04:48:46.344091 7136 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: tablet copy: Starting download of 0 data blocks...
I20260928 04:48:46.344254 7136 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: tablet copy: Starting download of 1 WAL segments...
I20260928 04:48:46.345098 7136 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 04:48:46.345880 6504 tablet_copy_service.cc:342] P c1d7afa74ea042a0bfcd16202957fa7d: Request end of tablet copy session ce55491dfcb148e7b3515700c9c0620a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48506
I20260928 04:48:46.345944 6504 tablet_copy_service.cc:433] P c1d7afa74ea042a0bfcd16202957fa7d: ending tablet copy session ce55491dfcb148e7b3515700c9c0620a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ce55491dfcb148e7b3515700c9c0620a
I20260928 04:48:46.348412 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:46.349514 7241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:46.349581 7244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:46.349669 7242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:46.353335 7136 server_base.cc:1034] running on GCE node
I20260928 04:48:46.353433 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:46.353603 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:46.354719 7136 hybrid_clock.cc:648] HybridClock initialized: now 1790570926354686 us; error 51 us; skew 500 ppm
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:46.355957 7136 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 04:48:46.356217 7136 webserver.cc:460] Webserver started at http://127.0.98.60:42601/ using document root <none> and password file <none>
I20260928 04:48:46.356339 7136 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:46.356372 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:46.356931 7136 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.357549 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.357697 7136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.357760 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal
uuid: "ce55491dfcb148e7b3515700c9c0620a"
format_stamp: "Formatted at 2026-09-28 04:48:46 on dist-test-slave-jccs"
server_key: "1699d94d31fcf5429c8933cba1a99398"
server_key_iv: "4a7bdd848eebf1a3cf6f0debcba6a426"
server_key_version: "encryptionkey@0"
I20260928 04:48:46.357836 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:46.371481 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:46.373122 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:46.376345 7136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37139
I20260928 04:48:46.376425 7303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37139 every 8 connection(s)
I20260928 04:48:46.376636 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
I20260928 04:48:46.377422 7304 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:48:46.377861 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: Bootstrap starting.
I20260928 04:48:46.380713 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:48:46.380936 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: Bootstrap complete.
I20260928 04:48:46.381114 7304 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } }
I20260928 04:48:46.381222 7304 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ce55491dfcb148e7b3515700c9c0620a, State: Initialized, Role: LEARNER
I20260928 04:48:46.381271 7304 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } }
I20260928 04:48:46.381424 7304 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:46.381435 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.381596 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: This master's current role is: LEARNER
I20260928 04:48:46.382915 7318 catalog_manager.cc:1259] Loaded cluster ID: eac889087153461c93f3e51a5e012818
I20260928 04:48:46.382965 7318 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: loading cluster ID for follower catalog manager: success
I20260928 04:48:46.383980 7318 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: acquiring CA information for follower catalog manager: success
I20260928 04:48:46.384233 7318 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:48:46.429710 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.423774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.0.98.60:37139)
0928 04:48:46.423907 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.423909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.423919 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:46.423946 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.423947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.423965 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.424046 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.424576 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.425153 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.425914 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.426102 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.426205 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.427343 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.427344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.427345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.427352 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.428671 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.428997 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.428998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.428998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.429053 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.429242 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.429243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.429243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.429349 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.429443 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.429536 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.429618 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260928 04:48:46.430182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 6546
I20260928 04:48:46.434244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40905
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43173
--webserver_port=36543
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43173,127.0.98.61:40337
--tserver_master_addrs=127.0.98.62:43173,127.0.98.61:40337,127.0.98.60:37139 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:46.510308 7320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:46.510478 7320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:46.510499 7320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:46.511521 7320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:46.511559 7320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:46.511605 7320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:46.512965 7320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36543
--tserver_master_addrs=127.0.98.62:43173,127.0.98.61:40337,127.0.98.60:37139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:46.513211 7320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:46.513478 7320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:46.517970 7328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:46.518129 7320 server_base.cc:1034] running on GCE node
W20260928 04:48:46.517970 7325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:46.517992 7326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:46.518443 7320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:46.518682 7320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:46.519820 7320 hybrid_clock.cc:648] HybridClock initialized: now 1790570926519814 us; error 25 us; skew 500 ppm
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:46.522080 7320 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 04:48:46.522444 7320 webserver.cc:460] Webserver started at http://127.0.98.1:36543/ using document root <none> and password file <none>
I20260928 04:48:46.522655 7320 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:46.522728 7320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:46.523923 7320 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:46.524483 7335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.524664 7320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:46.524755 7320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "14aef32c5a874dc39da827c6377d5e59"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "f7e278f6e587d011e2335c97bf2c1b30"
server_key_iv: "9bce8eea59778fa7b2334ed5c10ab652"
server_key_version: "encryptionkey@0"
I20260928 04:48:46.525051 7320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:46.549471 7320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:46.551609 7320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:46.551774 7320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:46.552004 7320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:46.552299 7320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:46.552356 7320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.552397 7320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:46.552423 7320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.558143 7320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40905
I20260928 04:48:46.558185 7448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40905 every 8 connection(s)
I20260928 04:48:46.558513 7320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 04:48:46.568518 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.559491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33607 (local address 127.0.98.60:37139)
0928 04:48:46.559534 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.559536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.559546 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:46.560135 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.560136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.560150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.560184 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.560900 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.561657 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.562495 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.562776 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.562851 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.565112 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.565114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.565114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.565122 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.566886 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.567422 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.567423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.567423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.567479 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.567754 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.567755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.567755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.567882 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.567914 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.568275 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.568392 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:48:46.568521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 7320
I20260928 04:48:46.568677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 6680
I20260928 04:48:46.568953 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.559461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48795 (local address 127.0.98.61:40337)
0928 04:48:46.559501 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.559503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.559672 (+ 169us) server_negotiation.cc:407] Connection header received
0928 04:48:46.560135 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.560136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.560150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.560184 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.560846 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.561407 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.562649 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.562919 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.562992 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.566119 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.566120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.566120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.566128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.567726 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.568074 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.568075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.568076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.568129 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.568339 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.568340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.568340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.568479 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.568521 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.568809 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.568884 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:48:46.569061 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.559168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.98.1:33607)
0928 04:48:46.559346 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.559400 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.560035 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.560631 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.560632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.560770 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.560831 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.560833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.561746 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.561747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.562407 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.562409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.562855 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.562856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.562996 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.563628 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.563633 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.565016 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.566948 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.566976 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.566978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.567294 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.567555 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.567555 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.567556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.567680 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.567950 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.567955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.568118 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.568625 (+ 507us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.568798 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":20,"thread_start_us":86,"threads_started":1}
I20260928 04:48:46.569274 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.559168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40337 (local address 127.0.98.1:48795)
0928 04:48:46.559602 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.559611 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.560035 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.560274 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.560275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.560600 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.560736 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.560742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.561510 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.561511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.562572 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.562573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.563197 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.563198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.563233 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.563939 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.563945 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.566013 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.567804 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.567805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.567805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.568021 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.568201 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.568203 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.568203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.568261 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.568582 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.568583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.568682 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.569046 (+ 364us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.569148 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":290,"thread_start_us":70,"threads_started":1}
I20260928 04:48:46.569674 7449 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37139
I20260928 04:48:46.569773 7449 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:46.569932 7449 heartbeater.cc:511] Master 127.0.98.60:37139 requested a full tablet report, sending...
I20260928 04:48:46.570390 7268 ts_manager.cc:194] Registered new tserver with Master: 14aef32c5a874dc39da827c6377d5e59 (127.0.98.1:40905)
I20260928 04:48:46.573082 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.559162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44305 (local address 127.0.98.62:43173)
0928 04:48:46.559209 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.559211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.559515 (+ 304us) server_negotiation.cc:407] Connection header received
0928 04:48:46.560114 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.560115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.560127 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.560184 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.560835 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.561627 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.562687 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.562915 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.562997 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.566430 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.566432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.566432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.566440 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.567934 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.568257 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.568258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.568258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.568309 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.572622 (+ 4313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.572623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.572623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.572761 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.572815 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.572968 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.573027 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:48:46.573231 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.559168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.1:44305)
0928 04:48:46.559346 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.559400 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.560009 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.560255 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.560258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.560559 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.560736 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.560740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.561705 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.561706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.562571 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.562573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.562996 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.562998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.563064 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.563560 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.563568 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.566302 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.568011 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.568012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.568012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.568201 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.572377 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.572378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.572379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.572450 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.572830 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.572831 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.572876 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.572954 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.573058 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":29,"thread_start_us":83,"threads_started":1}
I20260928 04:48:46.573611 7450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43173
I20260928 04:48:46.573661 7450 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:46.573732 7450 heartbeater.cc:511] Master 127.0.98.62:43173 requested a full tablet report, sending...
I20260928 04:48:46.573972 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40337
I20260928 04:48:46.574029 7451 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:46.574007 6484 ts_manager.cc:194] Re-registered known tserver with Master: 14aef32c5a874dc39da827c6377d5e59 (127.0.98.1:40905)
I20260928 04:48:46.574095 7451 heartbeater.cc:511] Master 127.0.98.61:40337 requested a full tablet report, sending...
I20260928 04:48:46.574442 7082 ts_manager.cc:194] Registered new tserver with Master: 14aef32c5a874dc39da827c6377d5e59 (127.0.98.1:40905)
I20260928 04:48:46.574501 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44305
I20260928 04:48:46.575614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37101
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43173
--webserver_port=45119
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43173,127.0.98.61:40337
--tserver_master_addrs=127.0.98.62:43173,127.0.98.61:40337,127.0.98.60:37139 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:46.658907 7456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:46.659168 7456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:46.659204 7456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:46.660318 7456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:46.660380 7456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:46.660424 7456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:46.661887 7456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45119
--tserver_master_addrs=127.0.98.62:43173,127.0.98.61:40337,127.0.98.60:37139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:46.662142 7456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:46.662366 7456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:46.667310 7461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:46.667464 7456 server_base.cc:1034] running on GCE node
W20260928 04:48:46.667315 7462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:46.667306 7464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:46.667768 7456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:46.668012 7456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:46.669163 7456 hybrid_clock.cc:648] HybridClock initialized: now 1790570926669143 us; error 32 us; skew 500 ppm
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:46.671787 7456 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 04:48:46.672205 7456 webserver.cc:460] Webserver started at http://127.0.98.2:45119/ using document root <none> and password file <none>
I20260928 04:48:46.672449 7456 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:46.672533 7456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:46.673992 7456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:46.674557 7471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.674726 7456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:46.674821 7456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "138d57a007d94250a5297be6e3b6fd04"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "ad6148c40682669873b028dd0e0f652c"
server_key_iv: "f3e81938e815dc15ccbf421db8628573"
server_key_version: "encryptionkey@0"
I20260928 04:48:46.675150 7456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:46.688985 7456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:46.691457 7456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:46.691625 7456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:46.691849 7456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:46.692149 7456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:46.692201 7456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.692241 7456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:46.692268 7456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.698071 7456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37101
I20260928 04:48:46.698117 7584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37101 every 8 connection(s)
I20260928 04:48:46.698567 7456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 04:48:46.706270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 7456
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:46.706444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 6814
I20260928 04:48:46.709328 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.699707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58331 (local address 127.0.98.61:40337)
0928 04:48:46.699745 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.699747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.699851 (+ 104us) server_negotiation.cc:407] Connection header received
0928 04:48:46.700428 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.700429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.700438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.700499 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.701272 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.701755 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.703042 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.703329 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.703402 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.705711 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.705712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.705712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.705718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.707699 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.708207 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.708208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.708209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.708269 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.708566 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.708567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.708568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.708713 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.708756 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.709124 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.709229 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:48:46.709573 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.699653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40337 (local address 127.0.98.2:58331)
0928 04:48:46.699767 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.699777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.700199 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.700530 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.700536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.700951 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.701134 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.701139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.701863 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.701864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.702968 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.702970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.703404 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.703405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.703468 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.704059 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.704064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.705626 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.707792 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.707798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.707801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.708121 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.708345 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.708346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.708346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.708483 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.708802 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.708807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.708983 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.709093 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.709302 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260928 04:48:46.709599 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.699319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40207 (local address 127.0.98.62:43173)
0928 04:48:46.699368 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.699370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.699653 (+ 283us) server_negotiation.cc:407] Connection header received
0928 04:48:46.700295 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.700296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.700310 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.700374 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.701276 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.702039 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.703239 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.703520 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.703592 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.706805 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.706807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.706807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.706814 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.708418 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.708738 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.708739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.708739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.708796 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.708995 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.708996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.708996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.709134 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.709214 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.709429 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.709521 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:48:46.709890 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.699331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.2:40207)
0928 04:48:46.699521 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.699567 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.700185 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.700878 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.700878 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.701026 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.701136 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.701142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.702425 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.702427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.703160 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.703162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.703645 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.703646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.703706 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.704222 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.704226 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.706711 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.708490 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.708491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.708492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.708659 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.708858 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.708859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.708860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.708924 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.709242 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.709243 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.709284 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.709671 (+ 387us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.709786 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":4,"spinlock_wait_cycles":6528,"thread_start_us":119,"threads_started":1}
I20260928 04:48:46.710358 7586 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43173
I20260928 04:48:46.710441 7586 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:46.710611 7586 heartbeater.cc:511] Master 127.0.98.62:43173 requested a full tablet report, sending...
I20260928 04:48:46.710942 6484 ts_manager.cc:194] Re-registered known tserver with Master: 138d57a007d94250a5297be6e3b6fd04 (127.0.98.2:37101)
I20260928 04:48:46.711436 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40207
I20260928 04:48:46.712631 7587 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40337
I20260928 04:48:46.712700 7587 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:46.712776 7587 heartbeater.cc:511] Master 127.0.98.61:40337 requested a full tablet report, sending...
I20260928 04:48:46.713109 7082 ts_manager.cc:194] Registered new tserver with Master: 138d57a007d94250a5297be6e3b6fd04 (127.0.98.2:37101)
I20260928 04:48:46.713786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42081
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43173
--webserver_port=38121
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43173,127.0.98.61:40337
--tserver_master_addrs=127.0.98.62:43173,127.0.98.61:40337,127.0.98.60:37139 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:48:46.716262 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.699774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37107 (local address 127.0.98.60:37139)
0928 04:48:46.699821 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.699823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.699832 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:46.700548 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.700550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.700560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.700610 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.701275 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.702034 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.702960 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.703246 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.703307 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.713568 (+ 10261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.713569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.713570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.713578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.715216 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.715611 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.715612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.715612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.715659 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.715850 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.715851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.715851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.715942 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.715990 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.716144 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.716203 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:48:46.716418 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.699338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.98.2:37107)
0928 04:48:46.699521 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.699567 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.700185 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.700634 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.700635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.700933 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.701136 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.701142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.702113 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.702114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.702859 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.702861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.703309 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.703310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.703458 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.704006 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.704013 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.706076 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.715295 (+ 9219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.715296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.715297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.715533 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.715722 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.715722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.715722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.715788 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.716002 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.716003 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.716044 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.716122 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.716234 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":55,"thread_start_us":111,"threads_started":1}
I20260928 04:48:46.716759 7585 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37139
I20260928 04:48:46.716811 7585 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:46.716866 7585 heartbeater.cc:511] Master 127.0.98.60:37139 requested a full tablet report, sending...
I20260928 04:48:46.717115 7268 ts_manager.cc:194] Registered new tserver with Master: 138d57a007d94250a5297be6e3b6fd04 (127.0.98.2:37101)
W20260928 04:48:46.794610 7592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:46.794823 7592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:46.794858 7592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:46.795984 7592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:46.796046 7592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:46.796089 7592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:46.797590 7592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38121
--tserver_master_addrs=127.0.98.62:43173,127.0.98.61:40337,127.0.98.60:37139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:46.797827 7592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:46.798059 7592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:46.802812 7597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:46.802835 7598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:46.803006 7592 server_base.cc:1034] running on GCE node
W20260928 04:48:46.803051 7600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:46.803260 7592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:46.803449 7592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:46.804581 7592 hybrid_clock.cc:648] HybridClock initialized: now 1790570926804568 us; error 30 us; skew 500 ppm
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:46.806999 7592 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 04:48:46.807392 7592 webserver.cc:460] Webserver started at http://127.0.98.3:38121/ using document root <none> and password file <none>
I20260928 04:48:46.807615 7592 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:46.807693 7592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:46.808945 7592 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:46.809516 7607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.809669 7592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.809763 7592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "79efdd818fc44a249647d6c2409d9d4d"
format_stamp: "Formatted at 2026-09-28 04:48:45 on dist-test-slave-jccs"
server_key: "8db35077349bbaacaf64bf9d63275e80"
server_key_iv: "ea4ba53289359b3391571ba7f8f9eca0"
server_key_version: "encryptionkey@0"
I20260928 04:48:46.810062 7592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:46.816548 7592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:46.818756 7592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:46.818910 7592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:46.819106 7592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:46.819379 7592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:46.819439 7592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.819478 7592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:46.819504 7592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:46.826078 7592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42081
I20260928 04:48:46.826118 7720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42081 every 8 connection(s)
I20260928 04:48:46.826473 7592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:48:46.828752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 7592
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 04:48:46 dist-test-slave-jccs krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:46.839157 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.827456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60715 (local address 127.0.98.60:37139)
0928 04:48:46.827527 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.827529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.827668 (+ 139us) server_negotiation.cc:407] Connection header received
0928 04:48:46.828522 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.828523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.828534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.828563 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.829567 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.830108 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.831139 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.831447 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.831488 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.835238 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.835240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.835240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.835247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.837104 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.838067 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.838069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.838070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.838126 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.838412 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.838413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.838413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.838556 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.838607 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.838976 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.839055 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:48:46.839701 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.827143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.98.3:60715)
0928 04:48:46.827611 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.827619 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.828420 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.828671 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.828672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.829153 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.829456 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.829463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.830239 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.830240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.831011 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.831013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.831576 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.831577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.831734 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.832268 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.832277 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.835134 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.837666 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.837672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.837674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.837974 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.838204 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.838204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.838205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.838327 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.838639 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.838645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.838805 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.839276 (+ 471us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.839466 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260928 04:48:46.840191 7721 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37139
I20260928 04:48:46.840286 7721 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:46.840468 7721 heartbeater.cc:511] Master 127.0.98.60:37139 requested a full tablet report, sending...
I20260928 04:48:46.840489 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.827388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43777 (local address 127.0.98.61:40337)
0928 04:48:46.827479 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.827481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.827548 (+ 67us) server_negotiation.cc:407] Connection header received
0928 04:48:46.828633 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.828635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.828645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.828691 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.829562 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.830320 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.831555 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.831853 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.831902 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.836352 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.836353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.836353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.836360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.838000 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.838581 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.838583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.838583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.838642 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.839855 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.839856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.839857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.840001 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.840046 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.840279 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.840358 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 04:48:46.840842 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.827124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40337 (local address 127.0.98.3:43777)
0928 04:48:46.827482 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.827493 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.828420 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.828733 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.828734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.829159 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.829454 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.829460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.830418 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.830419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.831483 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.831484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.831971 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.831971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.832034 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.832614 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.832618 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.836262 (+ 3644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.838082 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.838083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.838083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.838285 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.838755 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.838756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.838757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.838835 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.840098 (+ 1263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.840098 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.840144 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.840610 (+ 466us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.840731 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":72,"thread_start_us":92,"threads_started":1}
I20260928 04:48:46.840898 7268 ts_manager.cc:194] Registered new tserver with Master: 79efdd818fc44a249647d6c2409d9d4d (127.0.98.3:42081)
I20260928 04:48:46.841220 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.827385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37451 (local address 127.0.98.62:43173)
0928 04:48:46.827428 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.827430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.827440 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:46.828512 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.828513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.828527 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.828581 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.829562 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.830321 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.831445 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.831745 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.831789 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.835808 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.835809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.835810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.835818 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.838452 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.839085 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.839086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.839086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.839138 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.839985 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.839986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.839987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.840127 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.840187 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.840771 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.840846 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:48:46.841630 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.827123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.3:37451)
0928 04:48:46.827306 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.827345 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.828411 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.828647 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.828652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.829163 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:46.829454 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.829460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.830419 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.830420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.831320 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.831323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.831968 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.831969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.832047 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.832548 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:46.832553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:46.835713 (+ 3160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:46.838766 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.838767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.838767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.839009 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.839203 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:46.839204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:46.839205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:46.839707 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:46.840213 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:46.840214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:46.840253 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.840317 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.840415 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":51,"thread_start_us":81,"threads_started":1}
I20260928 04:48:46.841944 7723 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40337
I20260928 04:48:46.842015 7723 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:46.842084 7723 heartbeater.cc:511] Master 127.0.98.61:40337 requested a full tablet report, sending...
I20260928 04:48:46.842339 7082 ts_manager.cc:194] Registered new tserver with Master: 79efdd818fc44a249647d6c2409d9d4d (127.0.98.3:42081)
I20260928 04:48:46.842833 7722 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43173
I20260928 04:48:46.842885 7722 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:46.842959 7722 heartbeater.cc:511] Master 127.0.98.62:43173 requested a full tablet report, sending...
I20260928 04:48:46.843161 6484 ts_manager.cc:194] Re-registered known tserver with Master: 79efdd818fc44a249647d6c2409d9d4d (127.0.98.3:42081)
I20260928 04:48:46.843659 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37451
I20260928 04:48:46.843982 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.831316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.0.98.62:43173)
0928 04:48:46.833413 (+ 2097us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.833414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.833419 (+ 5us) server_negotiation.cc:407] Connection header received
0928 04:48:46.833434 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.833435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.833467 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.833536 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.834754 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.835446 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.837203 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.837545 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.837612 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.840895 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.840896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.840897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.840904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.842723 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.843124 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.843126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.843126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.843181 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.843414 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.843415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.843416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.843526 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.843628 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.843785 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.843882 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2083,"thread_start_us":44,"threads_started":1}
I20260928 04:48:46.845590 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.832471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.0.98.61:40337)
0928 04:48:46.833729 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.833732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.833739 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:48:46.833757 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.833758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.833769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.833823 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.834515 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.836873 (+ 2358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.838172 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.838486 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.838546 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.841437 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.841439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.841439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.841446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.843479 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.844689 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.844690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.844690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.844740 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.844957 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.844957 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.844958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.845067 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.845145 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.845365 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.845525 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1241,"thread_start_us":55,"threads_started":1}
I20260928 04:48:46.846109 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.831850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.0.98.60:37139)
0928 04:48:46.832639 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.832641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.833356 (+ 715us) server_negotiation.cc:407] Connection header received
0928 04:48:46.833383 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.833384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.833395 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.833471 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:46.834365 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.835092 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.836619 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.837306 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.840944 (+ 3638us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.840986 (+ 42us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:46.840987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:46.840987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:46.840991 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:46.844192 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.844629 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.844631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.844631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.844687 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:46.844914 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:46.844915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:46.844915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:46.845034 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:46.845133 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.845951 (+ 818us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.846027 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":54,"threads_started":1}
I20260928 04:48:46.845547 6484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:46.847429 6484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:46.854712 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.850898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42081 (local address 127.0.0.1:59626)
0928 04:48:46.851083 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.851094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.851132 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.851472 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.851473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.851476 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:46.851518 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.851520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.852332 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.852333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.853594 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.853596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.854482 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.854483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.854550 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.854553 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.854585 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.854632 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":39,"thread_start_us":82,"threads_started":1}
I20260928 04:48:46.854720 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.850800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37101 (local address 127.0.0.1:41218)
0928 04:48:46.850929 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.850940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.850984 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.851546 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.851547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.851549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:46.851591 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.851593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.852394 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.852395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.853578 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.853580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.854469 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.854470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.854536 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.854540 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.854592 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.854636 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":26,"thread_start_us":86,"threads_started":1}
I20260928 04:48:46.855008 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.850925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.0.98.2:37101)
0928 04:48:46.851367 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.851371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.851381 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:46.851433 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.851435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.851449 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.851660 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:46.851717 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.852334 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.853678 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.854380 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.854458 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.854531 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.854602 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.854644 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":88,"threads_started":1}
I20260928 04:48:46.855049 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.851154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.0.98.3:42081)
0928 04:48:46.851287 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.851292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.851302 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:46.851338 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.851339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.851356 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.851472 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:46.851578 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.852245 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.853677 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.854383 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.854834 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.854904 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.854928 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.854966 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260928 04:48:46.855659 7519 tablet_service.cc:1467] Processing CreateTablet for tablet 8120a532c9ce43f0b7f62b23eacc4bde (DEFAULT_TABLE table=test-table [id=ec78eee0cb4142338ae0d26b572c26b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:46.855872 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.850988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40905 (local address 127.0.0.1:45814)
0928 04:48:46.852431 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.852439 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.852507 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.852691 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.852692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.852694 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:46.852740 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.852742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.853646 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.853647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.854442 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.854444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.855623 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.855624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.855687 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.855691 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.855759 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.855799 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"thread_start_us":66,"threads_started":1}
I20260928 04:48:46.855806 7655 tablet_service.cc:1467] Processing CreateTablet for tablet 8120a532c9ce43f0b7f62b23eacc4bde (DEFAULT_TABLE table=test-table [id=ec78eee0cb4142338ae0d26b572c26b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:46.856022 7519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8120a532c9ce43f0b7f62b23eacc4bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:46.856037 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.851184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.0.98.1:40905)
0928 04:48:46.851357 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.851361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.852506 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 04:48:46.852578 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.852580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.852595 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.852693 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:46.852793 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.853560 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.854537 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.855526 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.855605 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.855670 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.855774 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.855815 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260928 04:48:46.856169 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8120a532c9ce43f0b7f62b23eacc4bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:46.856658 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 8120a532c9ce43f0b7f62b23eacc4bde (DEFAULT_TABLE table=test-table [id=ec78eee0cb4142338ae0d26b572c26b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:46.856947 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8120a532c9ce43f0b7f62b23eacc4bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:46.859313 7749 tablet_bootstrap.cc:492] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: Bootstrap starting.
I20260928 04:48:46.859926 7749 tablet_bootstrap.cc:654] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:46.860136 7751 tablet_bootstrap.cc:492] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: Bootstrap starting.
I20260928 04:48:46.860296 7749 log.cc:824] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:46.860358 7750 tablet_bootstrap.cc:492] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: Bootstrap starting.
I20260928 04:48:46.860953 7749 tablet_bootstrap.cc:492] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: No bootstrap required, opened a new log
I20260928 04:48:46.861034 7749 ts_tablet_manager.cc:1397] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:46.861299 7751 tablet_bootstrap.cc:654] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:46.861466 7750 tablet_bootstrap.cc:654] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:46.861629 7751 log.cc:824] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:46.861728 7750 log.cc:824] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:46.862329 7751 tablet_bootstrap.cc:492] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: No bootstrap required, opened a new log
I20260928 04:48:46.862385 7751 ts_tablet_manager.cc:1397] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:48:46.862936 7749 raft_consensus.cc:348] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } }
I20260928 04:48:46.863078 7749 raft_consensus.cc:374] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:46.863121 7749 raft_consensus.cc:729] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138d57a007d94250a5297be6e3b6fd04, State: Initialized, Role: FOLLOWER
I20260928 04:48:46.863235 7749 consensus_queue.cc:260] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } }
I20260928 04:48:46.863488 7586 heartbeater.cc:503] Master 127.0.98.62:43173 was elected leader, sending a full tablet report...
I20260928 04:48:46.863843 7749 ts_tablet_manager.cc:1428] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:48:46.864109 7751 raft_consensus.cc:348] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } }
I20260928 04:48:46.864254 7751 raft_consensus.cc:374] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:46.864293 7751 raft_consensus.cc:729] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14aef32c5a874dc39da827c6377d5e59, State: Initialized, Role: FOLLOWER
I20260928 04:48:46.864399 7751 consensus_queue.cc:260] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } }
I20260928 04:48:46.864660 7751 ts_tablet_manager.cc:1428] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:48:46.864701 7450 heartbeater.cc:503] Master 127.0.98.62:43173 was elected leader, sending a full tablet report...
I20260928 04:48:46.865415 7750 tablet_bootstrap.cc:492] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: No bootstrap required, opened a new log
I20260928 04:48:46.865481 7750 ts_tablet_manager.cc:1397] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 04:48:46.867297 7750 raft_consensus.cc:348] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } }
I20260928 04:48:46.867414 7750 raft_consensus.cc:374] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:46.867435 7750 raft_consensus.cc:729] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79efdd818fc44a249647d6c2409d9d4d, State: Initialized, Role: FOLLOWER
I20260928 04:48:46.867534 7750 consensus_queue.cc:260] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } }
I20260928 04:48:46.867746 7750 ts_tablet_manager.cc:1428] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 04:48:46.867825 7722 heartbeater.cc:503] Master 127.0.98.62:43173 was elected leader, sending a full tablet report...
I20260928 04:48:46.897678 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.895223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.0.98.60:37139)
0928 04:48:46.895281 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:46.895283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:46.895301 (+ 18us) server_negotiation.cc:407] Connection header received
0928 04:48:46.895327 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:46.895327 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:46.895352 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:46.895430 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:46.895562 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.896042 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.896928 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.897380 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.897460 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.897535 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:46.897573 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 04:48:46.897609 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:48:46.897678 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:46.895104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.0.1:32870)
0928 04:48:46.895163 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:46.895176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:46.895255 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:46.895432 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:46.895433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:46.895435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:46.895494 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.895496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.896121 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.896122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.896820 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:46.896822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:46.897457 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:46.897459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:46.897516 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:46.897519 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:46.897567 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 04:48:46.897613 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:48:46.898051 7278 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 04:48:46.898886 7758 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1d7afa74ea042a0bfcd16202957fa7d. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:46.898995 7758 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: This master's current role is: LEARNER
W20260928 04:48:46.950347 7588 tablet.cc:2369] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:48:47.008513 7756 raft_consensus.cc:482] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:47.008623 7756 raft_consensus.cc:504] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } }
I20260928 04:48:47.009124 7756 leader_election.cc:288] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 138d57a007d94250a5297be6e3b6fd04 (127.0.98.2:37101), 79efdd818fc44a249647d6c2409d9d4d (127.0.98.3:42081)
I20260928 04:48:47.012785 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:47.009399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41855 (local address 127.0.98.3:42081)
0928 04:48:47.009453 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:47.009455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:47.009464 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:47.009492 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:47.009493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:47.009503 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:47.009574 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:47.009764 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.010484 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.011921 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.012422 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.012506 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:47.012516 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:47.012688 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 04:48:47.012723 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:48:47.012785 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:47.009399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51589 (local address 127.0.98.2:37101)
0928 04:48:47.009451 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:47.009454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:47.009464 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:47.009493 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:47.009493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:47.009505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:47.009574 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:47.009767 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.010480 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.011949 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.012448 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.012506 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:47.012516 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:47.012684 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 04:48:47.012720 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:48:47.012801 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:47.009315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42081 (local address 127.0.98.1:41855)
0928 04:48:47.009386 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:47.009393 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:47.009423 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:47.009589 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:47.009590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:47.009593 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:47.009689 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:47.009690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.010560 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.010560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:47.011837 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:47.011839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.012527 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.012528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:47.012610 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:47.012616 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:48:47.012668 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:48:47.012714 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":153}
I20260928 04:48:47.012828 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:47.009315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37101 (local address 127.0.98.1:51589)
0928 04:48:47.009353 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:47.009364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:47.009406 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:47.009589 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:47.009590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:47.009591 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:47.009689 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:47.009691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.010560 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.010560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:47.011837 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:47.011839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.012527 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.012528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:47.012610 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:47.012617 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 04:48:47.012668 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:48:47.012714 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":69}
I20260928 04:48:47.013126 7675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8120a532c9ce43f0b7f62b23eacc4bde" candidate_uuid: "14aef32c5a874dc39da827c6377d5e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79efdd818fc44a249647d6c2409d9d4d" is_pre_election: true
I20260928 04:48:47.013288 7675 raft_consensus.cc:2454] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14aef32c5a874dc39da827c6377d5e59 in term 0.
I20260928 04:48:47.013294 7539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8120a532c9ce43f0b7f62b23eacc4bde" candidate_uuid: "14aef32c5a874dc39da827c6377d5e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138d57a007d94250a5297be6e3b6fd04" is_pre_election: true
I20260928 04:48:47.013407 7539 raft_consensus.cc:2454] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14aef32c5a874dc39da827c6377d5e59 in term 0.
I20260928 04:48:47.013511 7338 leader_election.cc:302] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aef32c5a874dc39da827c6377d5e59, 79efdd818fc44a249647d6c2409d9d4d; no voters:
I20260928 04:48:47.013639 7756 raft_consensus.cc:2781] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:47.013688 7756 raft_consensus.cc:482] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:47.013716 7756 raft_consensus.cc:3037] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:47.014341 7756 raft_consensus.cc:504] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } }
I20260928 04:48:47.014475 7756 leader_election.cc:288] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [CANDIDATE]: Term 1 election: Requested vote from peers 138d57a007d94250a5297be6e3b6fd04 (127.0.98.2:37101), 79efdd818fc44a249647d6c2409d9d4d (127.0.98.3:42081)
I20260928 04:48:47.014596 7539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8120a532c9ce43f0b7f62b23eacc4bde" candidate_uuid: "14aef32c5a874dc39da827c6377d5e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138d57a007d94250a5297be6e3b6fd04"
I20260928 04:48:47.014622 7675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8120a532c9ce43f0b7f62b23eacc4bde" candidate_uuid: "14aef32c5a874dc39da827c6377d5e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79efdd818fc44a249647d6c2409d9d4d"
I20260928 04:48:47.014667 7539 raft_consensus.cc:3037] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:47.014721 7675 raft_consensus.cc:3037] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:47.015259 7539 raft_consensus.cc:2454] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14aef32c5a874dc39da827c6377d5e59 in term 1.
I20260928 04:48:47.015434 7337 leader_election.cc:302] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138d57a007d94250a5297be6e3b6fd04, 14aef32c5a874dc39da827c6377d5e59; no voters:
I20260928 04:48:47.015453 7675 raft_consensus.cc:2454] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14aef32c5a874dc39da827c6377d5e59 in term 1.
I20260928 04:48:47.015540 7756 raft_consensus.cc:2781] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:47.015695 7756 raft_consensus.cc:686] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 1 LEADER]: Becoming Leader. State: Replica: 14aef32c5a874dc39da827c6377d5e59, State: Running, Role: LEADER
I20260928 04:48:47.015810 7756 consensus_queue.cc:237] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } }
I20260928 04:48:47.016569 6484 catalog_manager.cc:5495] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14aef32c5a874dc39da827c6377d5e59 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "14aef32c5a874dc39da827c6377d5e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } health_report { overall_health: UNKNOWN } } }
I20260928 04:48:47.017154 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: true } } }
I20260928 04:48:47.017280 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:47.017985 7232 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d: attempting to promote NON_VOTER ce55491dfcb148e7b3515700c9c0620a to VOTER
I20260928 04:48:47.018107 7232 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } }
I20260928 04:48:47.018476 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Refusing update from remote peer c1d7afa74ea042a0bfcd16202957fa7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 04:48:47.018560 7278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 LEARNER]: Refusing update from remote peer c1d7afa74ea042a0bfcd16202957fa7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 04:48:47.018697 7766 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 04:48:47.018774 7765 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 04:48:47.019315 7232 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ce55491dfcb148e7b3515700c9c0620a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:47.019452 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ce55491dfcb148e7b3515700c9c0620a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:47.019490 7278 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ce55491dfcb148e7b3515700c9c0620a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:47.020217 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:47.020344 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:47.020298 7739 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:47.020395 7739 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:47.020462 7739 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:47.020510 7739 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:47.020484 7758 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:47.020548 7758 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:48:47.020546 7768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:47.020620 7768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:48:47.038094 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:47.035944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.0.98.1:40905)
0928 04:48:47.035994 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:47.035995 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:47.036003 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:47.036055 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:47.036056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:47.036075 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:47.036139 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:47.036228 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.036693 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.037456 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.037638 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.037688 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:47.037799 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:47.037888 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:47.037940 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:47.037967 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 04:48:47.037983 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:48:47.039799 7539 raft_consensus.cc:1261] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04 [term 1 FOLLOWER]: Refusing update from remote peer 14aef32c5a874dc39da827c6377d5e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:47.039896 7675 raft_consensus.cc:1261] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d [term 1 FOLLOWER]: Refusing update from remote peer 14aef32c5a874dc39da827c6377d5e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:47.040174 7763 consensus_queue.cc:1035] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:47.040279 7756 consensus_queue.cc:1035] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:47.042948 7772 mvcc.cc:205] Tried to move back new op lower bound from 7334178517152366592 to 7334178517057146880. Current Snapshot: MvccSnapshot[applied={T|T < 7334178517152366592}]
I20260928 04:48:47.044190 7771 mvcc.cc:205] Tried to move back new op lower bound from 7334178517152366592 to 7334178517057146880. Current Snapshot: MvccSnapshot[applied={T|T < 7334178517152366592}]
I20260928 04:48:47.045336 6483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:47.045429 6483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:47.046581 6483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1d7afa74ea042a0bfcd16202957fa7d: Sending DeleteTablet for 3 replicas of tablet 8120a532c9ce43f0b7f62b23eacc4bde
I20260928 04:48:47.046903 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet 8120a532c9ce43f0b7f62b23eacc4bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45814
I20260928 04:48:47.046912 7519 tablet_service.cc:1514] Processing DeleteTablet for tablet 8120a532c9ce43f0b7f62b23eacc4bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41218
I20260928 04:48:47.047117 7779 tablet_replica.cc:330] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: stopping tablet replica
I20260928 04:48:47.047202 7779 raft_consensus.cc:2229] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:47.047291 7779 raft_consensus.cc:2258] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:47.047405 7655 tablet_service.cc:1514] Processing DeleteTablet for tablet 8120a532c9ce43f0b7f62b23eacc4bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59626
I20260928 04:48:47.047725 7779 ts_tablet_manager.cc:1905] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:47.047758 7781 tablet_replica.cc:330] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: stopping tablet replica
I20260928 04:48:47.047843 7781 raft_consensus.cc:2229] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:47.047948 7781 raft_consensus.cc:2258] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:47.048344 7781 ts_tablet_manager.cc:1905] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:47.048765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 6452
I20260928 04:48:47.049662 7779 ts_tablet_manager.cc:1918] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:47.049734 7779 log.cc:1196] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/8120a532c9ce43f0b7f62b23eacc4bde
I20260928 04:48:47.049971 7779 ts_tablet_manager.cc:1939] T 8120a532c9ce43f0b7f62b23eacc4bde P 138d57a007d94250a5297be6e3b6fd04: Deleting consensus metadata
I20260928 04:48:47.050655 7781 ts_tablet_manager.cc:1918] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:47.050710 7781 log.cc:1196] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/8120a532c9ce43f0b7f62b23eacc4bde
I20260928 04:48:47.050977 7781 ts_tablet_manager.cc:1939] T 8120a532c9ce43f0b7f62b23eacc4bde P 79efdd818fc44a249647d6c2409d9d4d: Deleting consensus metadata
I20260928 04:48:47.053297 7780 tablet_replica.cc:330] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: stopping tablet replica
I20260928 04:48:47.053426 7780 raft_consensus.cc:2229] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:47.053520 7780 raft_consensus.cc:2258] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:47.053925 7780 ts_tablet_manager.cc:1905] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:48:47.055398 7608 connection.cc:535] server connection from 127.0.0.1:59626 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 04:48:47.055595 7475 connection.cc:535] server connection from 127.0.0.1:41218 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:48:47.055763 7780 ts_tablet_manager.cc:1918] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:47.055821 7780 log.cc:1196] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/8120a532c9ce43f0b7f62b23eacc4bde
I20260928 04:48:47.056062 7780 ts_tablet_manager.cc:1939] T 8120a532c9ce43f0b7f62b23eacc4bde P 14aef32c5a874dc39da827c6377d5e59: Deleting consensus metadata
W20260928 04:48:47.056373 7338 connection.cc:441] server connection from 127.0.0.1:45814 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45814 (request call id 1) could send its response
I20260928 04:48:47.064457 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:47.057264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.0.98.60:37139)
0928 04:48:47.057303 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:47.057305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:47.057343 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:48:47.057440 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:47.057441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:47.057453 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:47.057501 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:47.058357 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.058876 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.059856 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.060168 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.060237 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:47.061634 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:47.061636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:47.061636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:47.061642 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:47.063337 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:47.063718 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:47.063719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:47.063720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:47.063779 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:47.063961 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:47.063962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:47.063962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:47.064078 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:47.064181 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:47.064281 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 04:48:47.064356 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:48:47.064985 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:47.057314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.0.98.61:40337)
0928 04:48:47.057359 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:47.057361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:47.057368 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:48:47.057391 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:47.057392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:47.057403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:47.057470 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:47.058362 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.059120 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.060019 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:47.060256 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:47.060315 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:47.062078 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:47.062079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:47.062080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:47.062087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:47.063645 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:47.064165 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:47.064166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:47.064167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:47.064218 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:47.064417 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:47.064418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:47.064418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:47.064552 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:47.064651 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:47.064797 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 04:48:47.064890 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:48:48.044163 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.043826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.2:46137)
0928 04:48:48.043978 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.044097 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":60,"threads_started":1}
I20260928 04:48:48.044167 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.043917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.1:51393)
0928 04:48:48.044017 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.044090 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":56,"threads_started":1}
W20260928 04:48:48.044428 7450 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
W20260928 04:48:48.044432 7586 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
I20260928 04:48:48.044860 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.044685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.3:33961)
0928 04:48:48.044779 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.044829 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":47,"threads_started":1}
W20260928 04:48:48.045047 7722 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43173 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
I20260928 04:48:48.546938 7792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1d7afa74ea042a0bfcd16202957fa7d)
I20260928 04:48:48.547088 7792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } }
I20260928 04:48:48.547370 7792 leader_election.cc:288] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173), ce55491dfcb148e7b3515700c9c0620a (127.0.98.60:37139)
I20260928 04:48:48.547613 7793 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1d7afa74ea042a0bfcd16202957fa7d)
I20260928 04:48:48.547654 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.547498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:42840)
0928 04:48:48.547583 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.547624 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":46,"threads_started":1}
I20260928 04:48:48.547742 7793 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } }
W20260928 04:48:48.547981 7067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
I20260928 04:48:48.548175 7793 leader_election.cc:288] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173), 33ed8f11d6494de087aa6ae314418ea5 (127.0.98.61:40337)
I20260928 04:48:48.548270 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.548059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:42856)
0928 04:48:48.548202 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.548238 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":42,"threads_started":1}
W20260928 04:48:48.548373 7253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
I20260928 04:48:48.548502 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.548420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:42864)
0928 04:48:48.548452 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.548480 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 04:48:48.548612 7067 leader_election.cc:334] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
I20260928 04:48:48.548875 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.548773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:42872)
0928 04:48:48.548820 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.548845 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:48:48.549019 7253 leader_election.cc:334] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
I20260928 04:48:48.551610 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.548186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.0.98.61:40337)
0928 04:48:48.548284 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:48.548286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:48.548396 (+ 110us) server_negotiation.cc:407] Connection header received
0928 04:48:48.548418 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:48.548418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:48.548427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:48.548489 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:48.548738 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:48.549450 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:48.550743 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:48.551270 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:48.551344 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:48.551354 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:48.551496 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 04:48:48.551536 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":79,"thread_start_us":66,"threads_started":1}
I20260928 04:48:48.551604 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.547725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.0.98.60:37139)
0928 04:48:48.547811 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:48.547813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:48.547823 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:48.547849 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:48.547850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:48.547859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:48.547930 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:48.548132 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:48.549348 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:48.550743 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:48.551279 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:48.551344 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:48.551358 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:48.551495 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 04:48:48.551534 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":35,"threads_started":1}
I20260928 04:48:48.551607 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.548107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40337 (local address 127.0.0.1:54682)
0928 04:48:48.548283 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.548292 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:48.548346 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:48.548551 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:48.548551 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:48.548553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:48.548605 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:48.548607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:48.549524 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:48.549525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:48.550658 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:48.550661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:48.551354 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:48.551355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:48.551415 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:48.551419 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:48.551484 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 04:48:48.551525 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":40,"threads_started":1}
I20260928 04:48:48.551625 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.547505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37139 (local address 127.0.0.1:58324)
0928 04:48:48.547718 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.547729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:48.547782 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:48.547943 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:48.547944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:48.547945 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:48.547997 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:48.547999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:48.549424 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:48.549425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:48.550658 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:48.550661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:48.551356 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:48.551357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:48.551415 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:48.551419 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:48.551484 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 04:48:48.551528 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":42,"threads_started":1}
I20260928 04:48:48.552084 7092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce55491dfcb148e7b3515700c9c0620a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33ed8f11d6494de087aa6ae314418ea5" is_pre_election: true
I20260928 04:48:48.552110 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33ed8f11d6494de087aa6ae314418ea5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ce55491dfcb148e7b3515700c9c0620a" is_pre_election: true
I20260928 04:48:48.552196 7092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce55491dfcb148e7b3515700c9c0620a in term 1.
I20260928 04:48:48.552186 7278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33ed8f11d6494de087aa6ae314418ea5 in term 1.
I20260928 04:48:48.552364 7252 leader_election.cc:302] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33ed8f11d6494de087aa6ae314418ea5, ce55491dfcb148e7b3515700c9c0620a; no voters: c1d7afa74ea042a0bfcd16202957fa7d
I20260928 04:48:48.552380 7068 leader_election.cc:302] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33ed8f11d6494de087aa6ae314418ea5, ce55491dfcb148e7b3515700c9c0620a; no voters: c1d7afa74ea042a0bfcd16202957fa7d
I20260928 04:48:48.552464 7792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 04:48:48.552477 7793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 04:48:48.552500 7792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1d7afa74ea042a0bfcd16202957fa7d)
I20260928 04:48:48.552505 7793 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1d7afa74ea042a0bfcd16202957fa7d)
I20260928 04:48:48.552523 7793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:48:48.552523 7792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:48:48.553153 7792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } }
I20260928 04:48:48.553159 7793 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } }
I20260928 04:48:48.553342 7793 leader_election.cc:288] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [CANDIDATE]: Term 2 election: Requested vote from peers c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173), 33ed8f11d6494de087aa6ae314418ea5 (127.0.98.61:40337)
I20260928 04:48:48.553333 7792 leader_election.cc:288] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 2 election: Requested vote from peers c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173), ce55491dfcb148e7b3515700c9c0620a (127.0.98.60:37139)
I20260928 04:48:48.553488 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.553390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:42882)
0928 04:48:48.553443 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.553466 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 04:48:48.553567 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33ed8f11d6494de087aa6ae314418ea5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ce55491dfcb148e7b3515700c9c0620a"
I20260928 04:48:48.553591 7092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce55491dfcb148e7b3515700c9c0620a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33ed8f11d6494de087aa6ae314418ea5"
I20260928 04:48:48.553653 7092 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ce55491dfcb148e7b3515700c9c0620a in current term 2: Already voted for candidate 33ed8f11d6494de087aa6ae314418ea5 in this term.
I20260928 04:48:48.553649 7278 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 33ed8f11d6494de087aa6ae314418ea5 in current term 2: Already voted for candidate ce55491dfcb148e7b3515700c9c0620a in this term.
I20260928 04:48:48.553761 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.553660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:42894)
0928 04:48:48.553717 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.553740 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 04:48:48.553654 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.553452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:42890)
0928 04:48:48.553485 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.553510 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 04:48:48.553874 7067 leader_election.cc:334] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
I20260928 04:48:48.553959 7067 leader_election.cc:302] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33ed8f11d6494de087aa6ae314418ea5; no voters: c1d7afa74ea042a0bfcd16202957fa7d, ce55491dfcb148e7b3515700c9c0620a
I20260928 04:48:48.554047 7792 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 04:48:48.554167 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:48.554069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:42898)
0928 04:48:48.554119 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:48.554146 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:48:48.554244 7253 leader_election.cc:334] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
I20260928 04:48:48.554289 7253 leader_election.cc:302] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce55491dfcb148e7b3515700c9c0620a; no voters: 33ed8f11d6494de087aa6ae314418ea5, c1d7afa74ea042a0bfcd16202957fa7d
I20260928 04:48:48.554364 7793 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 04:48:49.045006 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:49.044744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.2:49717)
0928 04:48:49.044904 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:49.044945 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260928 04:48:49.045061 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:49.044770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.1:36189)
0928 04:48:49.044966 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:49.045009 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":55,"threads_started":1}
I20260928 04:48:49.045608 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:49.045379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.3:44405)
0928 04:48:49.045498 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:49.045531 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260928 04:48:50.045868 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.045646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.2:53675)
0928 04:48:50.045780 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.045816 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260928 04:48:50.045857 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.045521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.1:42201)
0928 04:48:50.045709 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.045776 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
I20260928 04:48:50.046259 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.046040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.3:59423)
0928 04:48:50.046161 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.046198 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":52,"threads_started":1}
I20260928 04:48:50.159822 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:50.159926 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } }
I20260928 04:48:50.160126 7808 leader_election.cc:288] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173), ce55491dfcb148e7b3515700c9c0620a (127.0.98.60:37139)
I20260928 04:48:50.160364 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33ed8f11d6494de087aa6ae314418ea5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ce55491dfcb148e7b3515700c9c0620a" is_pre_election: true
I20260928 04:48:50.160467 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.160241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:43136)
0928 04:48:50.160378 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.160429 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260928 04:48:50.160508 7278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33ed8f11d6494de087aa6ae314418ea5 in term 2.
I20260928 04:48:50.160650 7068 leader_election.cc:302] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33ed8f11d6494de087aa6ae314418ea5, ce55491dfcb148e7b3515700c9c0620a; no voters:
I20260928 04:48:50.160805 7808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 04:48:50.160854 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.160752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:43144)
0928 04:48:50.160808 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.160833 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 04:48:50.160866 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:50.160928 7808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 2 FOLLOWER]: Advancing to term 3
W20260928 04:48:50.161032 7067 leader_election.cc:334] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
I20260928 04:48:50.161669 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } }
I20260928 04:48:50.161798 7808 leader_election.cc:288] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 3 election: Requested vote from peers c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173), ce55491dfcb148e7b3515700c9c0620a (127.0.98.60:37139)
I20260928 04:48:50.161963 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.161864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:43156)
0928 04:48:50.161915 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.161939 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 04:48:50.161942 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33ed8f11d6494de087aa6ae314418ea5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ce55491dfcb148e7b3515700c9c0620a"
I20260928 04:48:50.162019 7278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 2 FOLLOWER]: Advancing to term 3
I20260928 04:48:50.162263 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.162159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:43160)
0928 04:48:50.162214 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.162243 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 04:48:50.162340 7067 leader_election.cc:334] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
I20260928 04:48:50.162766 7278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ed8f11d6494de087aa6ae314418ea5 in term 3.
I20260928 04:48:50.162909 7068 leader_election.cc:302] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33ed8f11d6494de087aa6ae314418ea5, ce55491dfcb148e7b3515700c9c0620a; no voters: c1d7afa74ea042a0bfcd16202957fa7d
I20260928 04:48:50.163021 7808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 04:48:50.163141 7808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [term 3 LEADER]: Becoming Leader. State: Replica: 33ed8f11d6494de087aa6ae314418ea5, State: Running, Role: LEADER
I20260928 04:48:50.163228 7808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } }
I20260928 04:48:50.163586 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ed8f11d6494de087aa6ae314418ea5. Latest consensus state: current_term: 3 leader_uuid: "33ed8f11d6494de087aa6ae314418ea5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:50.163679 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:50.163836 7813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:50.164214 7813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:50.164265 7813 catalog_manager.cc:1259] Loaded cluster ID: eac889087153461c93f3e51a5e012818
I20260928 04:48:50.164281 7813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:50.164412 7813 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:50.164462 7813 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Loaded TSK: 0
I20260928 04:48:50.164558 7813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:48:50.220031 7081 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:50.221632 7081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:50.222604 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.222493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:43180)
0928 04:48:50.222526 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.222575 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 04:48:50.222615 7278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [term 3 FOLLOWER]: Refusing update from remote peer 33ed8f11d6494de087aa6ae314418ea5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 04:48:50.222807 7808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 04:48:50.222932 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.222846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.0.1:43188)
0928 04:48:50.222876 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.222901 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 04:48:50.223060 7067 consensus_peers.cc:487] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 -> Peer c1d7afa74ea042a0bfcd16202957fa7d (127.0.98.62:43173): Couldn't send request to peer c1d7afa74ea042a0bfcd16202957fa7d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 04:48:50.223412 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ed8f11d6494de087aa6ae314418ea5. Latest consensus state: current_term: 3 leader_uuid: "33ed8f11d6494de087aa6ae314418ea5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:50.223520 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:48:50.223963 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "33ed8f11d6494de087aa6ae314418ea5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:50.224050 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:50.224035 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "33ed8f11d6494de087aa6ae314418ea5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c1d7afa74ea042a0bfcd16202957fa7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43173 } } peers { permanent_uuid: "33ed8f11d6494de087aa6ae314418ea5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40337 } attrs { promote: false } } peers { permanent_uuid: "ce55491dfcb148e7b3515700c9c0620a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37139 } attrs { promote: false } } }
I20260928 04:48:50.224108 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce55491dfcb148e7b3515700c9c0620a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:48:50.224195 7816 mvcc.cc:205] Tried to move back new op lower bound from 7334178530189959168 to 7334178529949007872. Current Snapshot: MvccSnapshot[applied={T|T < 7334178530189959168 or (T in {7334178530189959168})}]
I20260928 04:48:50.229197 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.225842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42081 (local address 127.0.0.1:60130)
0928 04:48:50.225975 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.225984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:50.226029 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:50.226359 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:50.226360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:50.226362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:50.226411 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:50.226413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.227174 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.227176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:50.228070 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:50.228072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.228945 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.228946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:50.229004 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.229008 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:50.229065 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 04:48:50.229104 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":51,"thread_start_us":77,"threads_started":1}
I20260928 04:48:50.229219 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.226150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.0.98.3:42081)
0928 04:48:50.226243 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:50.226245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:50.226251 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:48:50.226270 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:50.226270 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:50.226279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:50.226357 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:50.226487 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.227111 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.228168 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.228867 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.228934 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.228947 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:50.229084 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 04:48:50.229120 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":54,"threads_started":1}
I20260928 04:48:50.229506 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.225899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40905 (local address 127.0.0.1:34032)
0928 04:48:50.226102 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.226109 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:50.226139 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:50.226307 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:50.226308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:50.226310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:50.226418 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:50.226420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.227356 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.227357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:50.228576 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:50.228578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.229279 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.229280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:50.229349 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.229354 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:48:50.229387 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 04:48:50.229431 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260928 04:48:50.229598 7655 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3f9b30f74a4e0191cb08a34f6d07e4 (DEFAULT_TABLE table=test-table [id=5462c839455247c198f19075d6ee342b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:50.229507 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.225809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37101 (local address 127.0.0.1:45422)
0928 04:48:50.225963 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.225974 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:50.226029 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:50.226475 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:50.226475 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:50.226477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:50.226504 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:50.226505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.227422 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.227423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:50.228576 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:50.228578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.229294 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.229295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:50.229365 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.229368 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:50.229396 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 04:48:50.229437 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":60}
I20260928 04:48:50.229673 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.225868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.0.98.2:37101)
0928 04:48:50.226265 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:50.226268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:50.226276 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:50.226308 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:50.226308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:50.226318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:50.226566 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:50.226601 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.227351 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.228645 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.229180 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.229546 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.229559 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:50.229576 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 04:48:50.229609 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":60,"threads_started":1}
I20260928 04:48:50.229735 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3f9b30f74a4e0191cb08a34f6d07e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:50.229887 7519 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3f9b30f74a4e0191cb08a34f6d07e4 (DEFAULT_TABLE table=test-table [id=5462c839455247c198f19075d6ee342b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:50.230007 7519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3f9b30f74a4e0191cb08a34f6d07e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:50.230038 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.226020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.0.98.1:40905)
0928 04:48:50.226139 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:50.226141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:50.226151 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:50.226189 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:50.226190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:50.226202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:50.226490 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:50.226533 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.227263 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.228675 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.229159 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.229893 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.229908 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:50.229926 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 04:48:50.229964 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":76,"threads_started":1}
I20260928 04:48:50.231014 7825 tablet_bootstrap.cc:492] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: Bootstrap starting.
I20260928 04:48:50.231199 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 3e3f9b30f74a4e0191cb08a34f6d07e4 (DEFAULT_TABLE table=test-table [id=5462c839455247c198f19075d6ee342b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:50.231325 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e3f9b30f74a4e0191cb08a34f6d07e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:50.231419 7825 tablet_bootstrap.cc:654] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:50.231658 7826 tablet_bootstrap.cc:492] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04: Bootstrap starting.
I20260928 04:48:50.231990 7826 tablet_bootstrap.cc:654] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:50.232043 7825 tablet_bootstrap.cc:492] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: No bootstrap required, opened a new log
I20260928 04:48:50.232084 7825 ts_tablet_manager.cc:1397] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:50.232196 7825 raft_consensus.cc:348] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } }
I20260928 04:48:50.232259 7825 raft_consensus.cc:374] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:50.232277 7825 raft_consensus.cc:729] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79efdd818fc44a249647d6c2409d9d4d, State: Initialized, Role: FOLLOWER
I20260928 04:48:50.232342 7825 consensus_queue.cc:260] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } }
I20260928 04:48:50.232510 7825 ts_tablet_manager.cc:1428] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:50.232558 7826 tablet_bootstrap.cc:492] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04: No bootstrap required, opened a new log
I20260928 04:48:50.232615 7826 ts_tablet_manager.cc:1397] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:50.232752 7826 raft_consensus.cc:348] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } }
I20260928 04:48:50.232831 7826 raft_consensus.cc:374] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:50.232852 7826 raft_consensus.cc:729] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138d57a007d94250a5297be6e3b6fd04, State: Initialized, Role: FOLLOWER
I20260928 04:48:50.232879 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.232792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.3:39493)
0928 04:48:50.232831 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.232854 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 04:48:50.232889 7826 consensus_queue.cc:260] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } }
I20260928 04:48:50.233050 7826 ts_tablet_manager.cc:1428] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:50.233345 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.233267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.2:53971)
0928 04:48:50.233302 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.233323 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 04:48:50.233500 7829 tablet_bootstrap.cc:492] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59: Bootstrap starting.
I20260928 04:48:50.233958 7829 tablet_bootstrap.cc:654] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:50.234063 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.233977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.2:60305)
0928 04:48:50.234014 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.234037 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 04:48:50.234227 7587 heartbeater.cc:503] Master 127.0.98.61:40337 was elected leader, sending a full tablet report...
I20260928 04:48:50.234375 7723 heartbeater.cc:503] Master 127.0.98.61:40337 was elected leader, sending a full tablet report...
I20260928 04:48:50.234443 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.234363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.3:51375)
0928 04:48:50.234397 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.234419 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 04:48:50.234831 7829 tablet_bootstrap.cc:492] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59: No bootstrap required, opened a new log
I20260928 04:48:50.234870 7829 ts_tablet_manager.cc:1397] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:50.234944 7829 raft_consensus.cc:348] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } }
I20260928 04:48:50.234992 7829 raft_consensus.cc:374] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:50.235004 7829 raft_consensus.cc:729] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14aef32c5a874dc39da827c6377d5e59, State: Initialized, Role: FOLLOWER
I20260928 04:48:50.235060 7829 consensus_queue.cc:260] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } }
I20260928 04:48:50.235206 7829 ts_tablet_manager.cc:1428] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:50.235448 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.235367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.1:43205)
0928 04:48:50.235398 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.235424 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 04:48:50.235807 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.235737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.1:52429)
0928 04:48:50.235764 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.235785 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 04:48:50.311734 7452 tablet.cc:2369] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:48:50.328720 7724 tablet.cc:2369] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:48:50.368096 7830 raft_consensus.cc:482] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:50.368196 7830 raft_consensus.cc:504] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } }
I20260928 04:48:50.368515 7830 leader_election.cc:288] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 138d57a007d94250a5297be6e3b6fd04 (127.0.98.2:37101), 14aef32c5a874dc39da827c6377d5e59 (127.0.98.1:40905)
I20260928 04:48:50.372332 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.368774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60883 (local address 127.0.98.2:37101)
0928 04:48:50.368827 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:50.368829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:50.368903 (+ 74us) server_negotiation.cc:407] Connection header received
0928 04:48:50.368984 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:50.368985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:50.368998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:50.369076 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:50.369266 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.369980 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.371427 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.371994 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.372066 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.372077 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:50.372227 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 04:48:50.372265 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:48:50.372340 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.368930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41217 (local address 127.0.98.1:40905)
0928 04:48:50.368972 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:50.368974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:50.369090 (+ 116us) server_negotiation.cc:407] Connection header received
0928 04:48:50.369163 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:50.369164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:50.369199 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:50.369275 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:50.369435 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.370043 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.371427 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.371988 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.372065 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.372077 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:50.372228 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 04:48:50.372265 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:48:50.372351 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.368715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37101 (local address 127.0.98.3:60883)
0928 04:48:50.368794 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.368807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:50.368901 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:50.369093 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:50.369094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:50.369097 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:50.369199 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:50.369200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.370052 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.370052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:50.371303 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:50.371306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.372079 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.372079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:50.372146 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.372150 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:50.372205 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:48:50.372262 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":56}
I20260928 04:48:50.372361 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.368849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40905 (local address 127.0.98.3:41217)
0928 04:48:50.369036 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.369045 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:50.369108 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:50.369337 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:50.369337 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:50.369339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:50.369381 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:50.369382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.370110 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.370111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:50.371303 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:50.371306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.372076 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.372077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:50.372146 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.372151 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:48:50.372205 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:48:50.372262 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":67,"thread_start_us":125,"threads_started":1}
I20260928 04:48:50.372684 7539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3f9b30f74a4e0191cb08a34f6d07e4" candidate_uuid: "79efdd818fc44a249647d6c2409d9d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138d57a007d94250a5297be6e3b6fd04" is_pre_election: true
I20260928 04:48:50.372681 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3f9b30f74a4e0191cb08a34f6d07e4" candidate_uuid: "79efdd818fc44a249647d6c2409d9d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14aef32c5a874dc39da827c6377d5e59" is_pre_election: true
I20260928 04:48:50.372771 7539 raft_consensus.cc:2454] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79efdd818fc44a249647d6c2409d9d4d in term 0.
I20260928 04:48:50.372803 7403 raft_consensus.cc:2454] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79efdd818fc44a249647d6c2409d9d4d in term 0.
I20260928 04:48:50.372962 7609 leader_election.cc:302] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aef32c5a874dc39da827c6377d5e59, 79efdd818fc44a249647d6c2409d9d4d; no voters:
I20260928 04:48:50.373117 7830 raft_consensus.cc:2781] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:50.373157 7830 raft_consensus.cc:482] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:50.373188 7830 raft_consensus.cc:3037] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:50.373806 7830 raft_consensus.cc:504] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } }
I20260928 04:48:50.373898 7830 leader_election.cc:288] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [CANDIDATE]: Term 1 election: Requested vote from peers 138d57a007d94250a5297be6e3b6fd04 (127.0.98.2:37101), 14aef32c5a874dc39da827c6377d5e59 (127.0.98.1:40905)
I20260928 04:48:50.374075 7539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3f9b30f74a4e0191cb08a34f6d07e4" candidate_uuid: "79efdd818fc44a249647d6c2409d9d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138d57a007d94250a5297be6e3b6fd04"
I20260928 04:48:50.374168 7539 raft_consensus.cc:3037] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:50.374790 7539 raft_consensus.cc:2454] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79efdd818fc44a249647d6c2409d9d4d in term 1.
I20260928 04:48:50.374914 7403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e3f9b30f74a4e0191cb08a34f6d07e4" candidate_uuid: "79efdd818fc44a249647d6c2409d9d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14aef32c5a874dc39da827c6377d5e59"
I20260928 04:48:50.374971 7609 leader_election.cc:302] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138d57a007d94250a5297be6e3b6fd04, 79efdd818fc44a249647d6c2409d9d4d; no voters:
I20260928 04:48:50.374989 7403 raft_consensus.cc:3037] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:50.375048 7830 raft_consensus.cc:2781] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:50.375202 7830 raft_consensus.cc:686] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 1 LEADER]: Becoming Leader. State: Replica: 79efdd818fc44a249647d6c2409d9d4d, State: Running, Role: LEADER
I20260928 04:48:50.375277 7830 consensus_queue.cc:237] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } }
I20260928 04:48:50.375768 7403 raft_consensus.cc:2454] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79efdd818fc44a249647d6c2409d9d4d in term 1.
I20260928 04:48:50.375959 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.375872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.3:55345)
0928 04:48:50.375905 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.375932 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 04:48:50.376127 7082 catalog_manager.cc:5495] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 79efdd818fc44a249647d6c2409d9d4d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "79efdd818fc44a249647d6c2409d9d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79efdd818fc44a249647d6c2409d9d4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 } health_report { overall_health: UNKNOWN } } }
I20260928 04:48:50.409135 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.407237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.0.98.3:42081)
0928 04:48:50.407294 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:50.407299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:50.407306 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:48:50.407336 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:50.407336 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:50.407354 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:50.407417 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:50.407519 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.407930 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.408555 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:50.408712 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:50.408758 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:50.408900 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:50.408987 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:50.409034 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:50.409070 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 04:48:50.409086 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:48:50.410181 7539 raft_consensus.cc:1261] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04 [term 1 FOLLOWER]: Refusing update from remote peer 79efdd818fc44a249647d6c2409d9d4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:50.410243 7403 raft_consensus.cc:1261] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59 [term 1 FOLLOWER]: Refusing update from remote peer 79efdd818fc44a249647d6c2409d9d4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:50.410426 7830 consensus_queue.cc:1035] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "138d57a007d94250a5297be6e3b6fd04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:50.410564 7835 consensus_queue.cc:1035] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "14aef32c5a874dc39da827c6377d5e59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:50.411167 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.411037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.2:47005)
0928 04:48:50.411083 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.411112 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 04:48:50.412070 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.411968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.1:51401)
0928 04:48:50.412003 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.412031 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 04:48:50.412735 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.412621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.3:33205)
0928 04:48:50.412686 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.412709 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 04:48:50.412897 7451 heartbeater.cc:503] Master 127.0.98.61:40337 was elected leader, sending a full tablet report...
I20260928 04:48:50.412930 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.412815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.2:45339)
0928 04:48:50.412877 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.412903 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 04:48:50.413273 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:50.413090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43173 (local address 127.0.98.1:59453)
0928 04:48:50.413206 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.413244 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 04:48:50.414469 7082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:50.414561 7082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:50.415752 7082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33ed8f11d6494de087aa6ae314418ea5: Sending DeleteTablet for 3 replicas of tablet 3e3f9b30f74a4e0191cb08a34f6d07e4
I20260928 04:48:50.416028 7655 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e3f9b30f74a4e0191cb08a34f6d07e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:50 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60130
I20260928 04:48:50.416100 7519 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e3f9b30f74a4e0191cb08a34f6d07e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:50 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45422
I20260928 04:48:50.416174 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e3f9b30f74a4e0191cb08a34f6d07e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:50 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34032
I20260928 04:48:50.416204 7850 tablet_replica.cc:330] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: stopping tablet replica
I20260928 04:48:50.416245 7850 raft_consensus.cc:2229] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:50.416316 7850 raft_consensus.cc:2258] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:50.416344 7852 tablet_replica.cc:330] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59: stopping tablet replica
I20260928 04:48:50.416385 7852 raft_consensus.cc:2229] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 14aef32c5a874dc39da827c6377d5e59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:50.416518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 7320
I20260928 04:48:50.417158 7850 ts_tablet_manager.cc:1905] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:50.418918 7850 ts_tablet_manager.cc:1918] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:50.418980 7850 log.cc:1196] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/3e3f9b30f74a4e0191cb08a34f6d07e4
I20260928 04:48:50.419199 7850 ts_tablet_manager.cc:1939] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 79efdd818fc44a249647d6c2409d9d4d: Deleting consensus metadata
I20260928 04:48:50.419353 7851 tablet_replica.cc:330] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04: stopping tablet replica
I20260928 04:48:50.419436 7851 raft_consensus.cc:2229] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:50.419502 7851 raft_consensus.cc:2258] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:50.419579 7068 catalog_manager.cc:4841] TS 79efdd818fc44a249647d6c2409d9d4d (127.0.98.3:42081): tablet 3e3f9b30f74a4e0191cb08a34f6d07e4 (table test-table [id=5462c839455247c198f19075d6ee342b]) successfully deleted
I20260928 04:48:50.419836 7851 ts_tablet_manager.cc:1905] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:50.421545 7851 ts_tablet_manager.cc:1918] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:50.421604 7851 log.cc:1196] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/3e3f9b30f74a4e0191cb08a34f6d07e4
I20260928 04:48:50.421824 7851 ts_tablet_manager.cc:1939] T 3e3f9b30f74a4e0191cb08a34f6d07e4 P 138d57a007d94250a5297be6e3b6fd04: Deleting consensus metadata
I20260928 04:48:50.422083 7067 catalog_manager.cc:4841] TS 138d57a007d94250a5297be6e3b6fd04 (127.0.98.2:37101): tablet 3e3f9b30f74a4e0191cb08a34f6d07e4 (table test-table [id=5462c839455247c198f19075d6ee342b]) successfully deleted
W20260928 04:48:50.423774 7822 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:50.423661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40905 (local address 127.0.0.1:34040)
0928 04:48:50.423710 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:50.423734 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40905: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 04:48:50.423928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 7456
W20260928 04:48:50.423976 7067 catalog_manager.cc:4584] TS 14aef32c5a874dc39da827c6377d5e59 (127.0.98.1:40905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e3f9b30f74a4e0191cb08a34f6d07e4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40905: connect: Connection reset by peer (error 104)
I20260928 04:48:50.428905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 7592
I20260928 04:48:50.433758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 6948
I20260928 04:48:50.438508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 7136
2026-09-28T04:48:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6291 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:50 dist-test-slave-jccs krb5kdc[7855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:50 dist-test-slave-jccs krb5kdc[7855](info): set up 2 sockets
Sep 28 04:48:50 dist-test-slave-jccs krb5kdc[7855](info): commencing operation
krb5kdc: starting...
W20260928 04:48:51.433651 392 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:51 dist-test-slave-jccs krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 04:48:51.547518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:51.625597 7868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:51.625852 7868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:51.625906 7868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:51.627072 7868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:51.627142 7868 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 04:48:51.627166 7868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:51.627198 7868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:51.627226 7868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:51.627249 7868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:51.628680 7868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39465
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:51.628948 7868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:51.629163 7868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:51.633850 7873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:51.633848 7868 server_base.cc:1034] running on GCE node
W20260928 04:48:51.633980 7874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:51.633942 7876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:51.634361 7868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:51.634696 7868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:51.635857 7868 hybrid_clock.cc:648] HybridClock initialized: now 1790570931635850 us; error 35 us; skew 500 ppm
Sep 28 04:48:51 dist-test-slave-jccs krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:51.638267 7868 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 04:48:51.638707 7868 webserver.cc:460] Webserver started at http://127.0.98.62:36605/ using document root <none> and password file <none>
I20260928 04:48:51.638902 7868 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:51.638942 7868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:51.639034 7868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:51.640025 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "89a2786edeb64703a90df4e9c4d26626"
format_stamp: "Formatted at 2026-09-28 04:48:51 on dist-test-slave-jccs"
server_key: "91b545af3b0c2c68111d35b240a92385"
server_key_iv: "16c9d62ee83acdc32a65fcd73503a44b"
server_key_version: "encryptionkey@0"
I20260928 04:48:51.640316 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "89a2786edeb64703a90df4e9c4d26626"
format_stamp: "Formatted at 2026-09-28 04:48:51 on dist-test-slave-jccs"
server_key: "91b545af3b0c2c68111d35b240a92385"
server_key_iv: "16c9d62ee83acdc32a65fcd73503a44b"
server_key_version: "encryptionkey@0"
I20260928 04:48:51.641491 7868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:48:51.642102 7883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:51.642285 7868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:51.642374 7868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "89a2786edeb64703a90df4e9c4d26626"
format_stamp: "Formatted at 2026-09-28 04:48:51 on dist-test-slave-jccs"
server_key: "91b545af3b0c2c68111d35b240a92385"
server_key_iv: "16c9d62ee83acdc32a65fcd73503a44b"
server_key_version: "encryptionkey@0"
I20260928 04:48:51.642442 7868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:51.655941 7868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:51.657863 7868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:51.658018 7868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:51.662426 7868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39465
I20260928 04:48:51.662473 7945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39465 every 8 connection(s)
I20260928 04:48:51.662772 7868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:51.663312 7946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:51.665520 7946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626: Bootstrap starting.
I20260928 04:48:51.666107 7946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:51.666384 7946 log.cc:824] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:51.666999 7946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626: No bootstrap required, opened a new log
I20260928 04:48:51.668948 7946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a2786edeb64703a90df4e9c4d26626" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260928 04:48:51.669097 7946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:51.669160 7946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a2786edeb64703a90df4e9c4d26626, State: Initialized, Role: FOLLOWER
I20260928 04:48:51.669335 7946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a2786edeb64703a90df4e9c4d26626" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260928 04:48:51.669415 7946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:51.669457 7946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:51.669504 7946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:51.670001 7946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a2786edeb64703a90df4e9c4d26626" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260928 04:48:51.670130 7946 leader_election.cc:302] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89a2786edeb64703a90df4e9c4d26626; no voters:
I20260928 04:48:51.670310 7946 leader_election.cc:288] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:51.670353 7949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:51.670552 7949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [term 1 LEADER]: Becoming Leader. State: Replica: 89a2786edeb64703a90df4e9c4d26626, State: Running, Role: LEADER
I20260928 04:48:51.670581 7946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:51.670730 7949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a2786edeb64703a90df4e9c4d26626" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } }
I20260928 04:48:51.671674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 7868
I20260928 04:48:51.671770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:51.671872 7950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89a2786edeb64703a90df4e9c4d26626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a2786edeb64703a90df4e9c4d26626" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } } }
I20260928 04:48:51.671957 7950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:51.671967 392 external_mini_cluster.cc:1428] Setting key bb9f6f85112606423b371f986a8309af
I20260928 04:48:51.672050 7951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89a2786edeb64703a90df4e9c4d26626. Latest consensus state: current_term: 1 leader_uuid: "89a2786edeb64703a90df4e9c4d26626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a2786edeb64703a90df4e9c4d26626" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39465 } } }
I20260928 04:48:51.672114 7951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:51.672165 7965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:51.672667 7965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:51.674072 7965 catalog_manager.cc:1347] Generated new cluster ID: 89b2cb3c42ab44d2b6379e4eb0b2e131
I20260928 04:48:51.674131 7965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:48:51 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:51.684296 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:51.672431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.0.98.62:39465)
0928 04:48:51.673287 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:51.673294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:51.673318 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:48:51.673409 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:51.673414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:51.673453 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:51.673620 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:51.674339 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.675681 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.676745 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.677365 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.677461 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:51.680227 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:51.680230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:51.680231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:51.680244 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:51.682341 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:51.682731 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:51.682734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:51.682734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:51.682853 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:51.683064 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:51.683065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:51.683065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:51.683368 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:51.683498 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:51.683927 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 04:48:51.684059 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":95,"threads_started":1}
I20260928 04:48:51.692128 7965 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:51.692644 7965 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:51.701498 7965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626: Generated new TSK 0
I20260928 04:48:51.701681 7965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 04:48:51.728354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39465
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:51.807161 7972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:51.807364 7972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:51.807386 7972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:51.808454 7972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:51.808485 7972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:51.808506 7972 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:48:51.808522 7972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:51.810029 7972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:51.810223 7972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:51.810382 7972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:51.815100 7977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:51.815238 7972 server_base.cc:1034] running on GCE node
W20260928 04:48:51.815179 7978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:51.815130 7980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:51.815578 7972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:51.815829 7972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:51.817022 7972 hybrid_clock.cc:648] HybridClock initialized: now 1790570931816997 us; error 30 us; skew 500 ppm
Sep 28 04:48:51 dist-test-slave-jccs krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:51.819540 7972 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 04:48:51.819968 7972 webserver.cc:460] Webserver started at http://127.0.98.1:41499/ using document root <none> and password file <none>
I20260928 04:48:51.820165 7972 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:51.820204 7972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:51.820288 7972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:51.821337 7972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "1bf275ca2b6f453f9f116f06ff6431bc"
format_stamp: "Formatted at 2026-09-28 04:48:51 on dist-test-slave-jccs"
server_key: "19752de0ed45bb2f77c0e327cadfaa4e"
server_key_iv: "a7b27899f56df5a9fa1103e08722b274"
server_key_version: "encryptionkey@0"
I20260928 04:48:51.821630 7972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1bf275ca2b6f453f9f116f06ff6431bc"
format_stamp: "Formatted at 2026-09-28 04:48:51 on dist-test-slave-jccs"
server_key: "19752de0ed45bb2f77c0e327cadfaa4e"
server_key_iv: "a7b27899f56df5a9fa1103e08722b274"
server_key_version: "encryptionkey@0"
I20260928 04:48:51.822798 7972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:51.823467 7987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:51.823634 7972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:51.823701 7972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "1bf275ca2b6f453f9f116f06ff6431bc"
format_stamp: "Formatted at 2026-09-28 04:48:51 on dist-test-slave-jccs"
server_key: "19752de0ed45bb2f77c0e327cadfaa4e"
server_key_iv: "a7b27899f56df5a9fa1103e08722b274"
server_key_version: "encryptionkey@0"
I20260928 04:48:51.823750 7972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:51.834874 7972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:51.837085 7972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:51.837252 7972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:51.837759 7972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:51.837800 7972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:51.837821 7972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:51.837833 7972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:48:51 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:51.845016 7972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39317
I20260928 04:48:51.845078 8102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39317 every 8 connection(s)
I20260928 04:48:51.845467 7972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:48:51.847566 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:51.838127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35315 (local address 127.0.98.62:39465)
0928 04:48:51.838208 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:51.838210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:51.838463 (+ 253us) server_negotiation.cc:407] Connection header received
0928 04:48:51.839174 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:51.839175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:51.839204 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:51.839278 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:51.840147 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.840611 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.842200 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.842452 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.842519 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:51.845019 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:51.845020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:51.845021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:51.845027 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:51.846530 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:51.846882 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:51.846882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:51.846883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:51.846930 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:51.847096 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:51.847097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:51.847097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:51.847190 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:51.847237 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:51.847438 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 04:48:51.847498 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:48:51.847752 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:51.838109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:35315)
0928 04:48:51.838310 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:51.838361 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:51.839071 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:51.839352 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:51.839356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:51.839766 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:51.840055 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:51.840061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.840686 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.840687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:51.842106 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:51.842109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.842557 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.842558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:51.842711 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:51.843188 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:51.843197 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:51.844887 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:51.846600 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:51.846605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:51.846607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:51.846813 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:51.846970 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:51.846970 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:51.846971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:51.847049 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:51.847243 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:51.847247 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:51.847350 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 04:48:51.847427 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 04:48:51.847534 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260928 04:48:51.848197 8103 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39465
I20260928 04:48:51.848269 8103 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:51.848434 8103 heartbeater.cc:511] Master 127.0.98.62:39465 requested a full tablet report, sending...
I20260928 04:48:51.848917 7899 ts_manager.cc:194] Registered new tserver with Master: 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317)
I20260928 04:48:51.849817 7899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35315
I20260928 04:48:51.852741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 7972
I20260928 04:48:51.852849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:51.853072 392 external_mini_cluster.cc:1428] Setting key 335f07cac76f91055deac90de0f58064
I20260928 04:48:51.854491 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:51.849172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.1:49945)
0928 04:48:51.849306 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:51.849317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:51.849375 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:51.849485 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:51.849485 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:51.849667 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:51.849718 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:51.849719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.850370 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.850370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:51.850897 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:51.850899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.851234 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.851235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:51.851279 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:51.851612 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:51.851616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:51.851871 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:51.853511 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:51.853512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:51.853512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:51.853718 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:51.853926 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:51.853926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:51.853927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:51.853992 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:51.854264 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:51.854265 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:51.854298 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 04:48:51.854347 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:48:51.854417 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":27,"thread_start_us":84,"threads_started":1}
I20260928 04:48:51.854583 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:51.849232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49945 (local address 127.0.98.62:39465)
0928 04:48:51.849270 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:51.849271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:51.849368 (+ 97us) server_negotiation.cc:407] Connection header received
0928 04:48:51.849425 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:51.849425 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:51.849432 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:51.849471 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:51.849777 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.850280 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.850987 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.851160 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.851234 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:51.851942 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:51.851943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:51.851944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:51.851949 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:51.853430 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:51.853785 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:51.853786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:51.853786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:51.853860 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:51.854066 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:51.854067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:51.854067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:51.854200 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:51.854267 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:51.854370 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 04:48:51.854451 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 04:48:51.876951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39465
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:51.952567 8112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:51.952745 8112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:51.952766 8112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:51.953754 8112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:51.953786 8112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:51.953804 8112 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:48:51.953820 8112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:51.955118 8112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:51.955287 8112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:51.955446 8112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:51.959987 8117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:51.959997 8120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:51.960191 8112 server_base.cc:1034] running on GCE node
W20260928 04:48:51.960022 8118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:51.960464 8112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:51.960709 8112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:51.961861 8112 hybrid_clock.cc:648] HybridClock initialized: now 1790570931961851 us; error 26 us; skew 500 ppm
Sep 28 04:48:51 dist-test-slave-jccs krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:51.964071 8112 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 04:48:51.964449 8112 webserver.cc:460] Webserver started at http://127.0.98.2:37511/ using document root <none> and password file <none>
I20260928 04:48:51.964680 8112 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:51.964747 8112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:51.964895 8112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:51.965986 8112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "600ab3a7dceb44ac877658947d3344d7"
format_stamp: "Formatted at 2026-09-28 04:48:51 on dist-test-slave-jccs"
server_key: "6c96362ad7402dea3385beb71c21f56b"
server_key_iv: "00eba52f512050d58b8f30333cb93231"
server_key_version: "encryptionkey@0"
I20260928 04:48:51.966338 8112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "600ab3a7dceb44ac877658947d3344d7"
format_stamp: "Formatted at 2026-09-28 04:48:51 on dist-test-slave-jccs"
server_key: "6c96362ad7402dea3385beb71c21f56b"
server_key_iv: "00eba52f512050d58b8f30333cb93231"
server_key_version: "encryptionkey@0"
I20260928 04:48:51.967572 8112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:48:51.968255 8127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:51.968415 8112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:51.968529 8112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "600ab3a7dceb44ac877658947d3344d7"
format_stamp: "Formatted at 2026-09-28 04:48:51 on dist-test-slave-jccs"
server_key: "6c96362ad7402dea3385beb71c21f56b"
server_key_iv: "00eba52f512050d58b8f30333cb93231"
server_key_version: "encryptionkey@0"
I20260928 04:48:51.968603 8112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:51.981664 8112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:51.983265 8112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:51.983426 8112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:51.983868 8112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:51.983925 8112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:51.983958 8112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:51.983997 8112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:51.991113 8112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42153
I20260928 04:48:51.991173 8242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42153 every 8 connection(s)
Sep 28 04:48:51 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:51.991508 8112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:48:51.994161 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:51.984329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58651 (local address 127.0.98.62:39465)
0928 04:48:51.984408 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:51.984417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:51.984684 (+ 267us) server_negotiation.cc:407] Connection header received
0928 04:48:51.985410 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:51.985411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:51.985438 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:51.985511 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:51.986448 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.987243 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.988799 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.989105 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.989210 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:51.991724 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:51.991725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:51.991725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:51.991734 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:51.993108 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:51.993483 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:51.993484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:51.993484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:51.993535 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:51.993708 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:51.993709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:51.993709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:51.993800 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:51.993847 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:51.994049 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 04:48:51.994102 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 04:48:51.994370 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:51.984308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:58651)
0928 04:48:51.984512 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:51.984564 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:51.985308 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:51.985586 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:51.985591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:51.986022 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:51.986330 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:51.986337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.987339 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.987340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:51.988708 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:51.988711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.989221 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.989222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:51.989367 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:51.989906 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:51.989916 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:51.991626 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:51.993212 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:51.993216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:51.993218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:51.993422 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:51.993585 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:51.993585 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:51.993585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:51.993662 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:51.993853 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:51.993858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:51.993953 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 04:48:51.994027 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 04:48:51.994132 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260928 04:48:51.994858 8243 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39465
I20260928 04:48:51.994935 8243 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:51.995106 8243 heartbeater.cc:511] Master 127.0.98.62:39465 requested a full tablet report, sending...
I20260928 04:48:51.995525 7900 ts_manager.cc:194] Registered new tserver with Master: 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2:42153)
I20260928 04:48:51.995987 7900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58651
I20260928 04:48:52.000484 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:51.995839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42829 (local address 127.0.98.62:39465)
0928 04:48:51.995873 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:51.995875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:51.995992 (+ 117us) server_negotiation.cc:407] Connection header received
0928 04:48:51.996104 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:51.996105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:51.996114 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:51.996168 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:51.996516 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.996946 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.997617 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.997784 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.997832 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:51.998652 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:51.998653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:51.998653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:51.998658 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:51.999700 (+ 1042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:51.999951 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:51.999952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:51.999952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:52.000011 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:52.000120 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:52.000120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:52.000120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:52.000226 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:52.000271 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.000375 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.000432 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 04:48:52.000490 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:51.995784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.2:42829)
0928 04:48:51.995922 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:51.995933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:51.996005 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:51.996172 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:51.996173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:51.996383 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:51.996445 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:51.996447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.997034 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.997034 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:51.997555 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:51.997556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:51.997843 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:51.997844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:51.997886 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:51.998315 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:51.998320 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:51.998581 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:51.999760 (+ 1179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:51.999761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:51.999761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:51.999896 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:52.000049 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:52.000049 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:52.000050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:52.000089 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:52.000284 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:52.000285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:52.000315 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.000360 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.000425 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260928 04:48:52.001206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 8112
I20260928 04:48:52.001315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:52.001513 392 external_mini_cluster.cc:1428] Setting key 46bc1c00fd6a07c019af949d360bdf41
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 04:48:52.024015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39465
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:52.102921 8252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:52.103107 8252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:52.103130 8252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:52.104250 8252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:52.104283 8252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:52.104303 8252 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 04:48:52.104321 8252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:52.105773 8252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:52.105973 8252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:52.106146 8252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:52.110762 8260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:52.110762 8258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:52.110773 8257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:52.110971 8252 server_base.cc:1034] running on GCE node
I20260928 04:48:52.111208 8252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:52.111393 8252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:52.112555 8252 hybrid_clock.cc:648] HybridClock initialized: now 1790570932112532 us; error 34 us; skew 500 ppm
Sep 28 04:48:52 dist-test-slave-jccs krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570932, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:52.114876 8252 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 04:48:52.115326 8252 webserver.cc:460] Webserver started at http://127.0.98.3:35479/ using document root <none> and password file <none>
I20260928 04:48:52.115558 8252 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:52.115625 8252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:52.115783 8252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:52.116756 8252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c292c19bceb425ba2a232bbc4978ca0"
format_stamp: "Formatted at 2026-09-28 04:48:52 on dist-test-slave-jccs"
server_key: "b62f09811d2b3799e39f6ec4850ae016"
server_key_iv: "c2f7e27ffe3d3875569494430fd00c7c"
server_key_version: "encryptionkey@0"
I20260928 04:48:52.117095 8252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c292c19bceb425ba2a232bbc4978ca0"
format_stamp: "Formatted at 2026-09-28 04:48:52 on dist-test-slave-jccs"
server_key: "b62f09811d2b3799e39f6ec4850ae016"
server_key_iv: "c2f7e27ffe3d3875569494430fd00c7c"
server_key_version: "encryptionkey@0"
I20260928 04:48:52.118362 8252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:52.119077 8267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:52.119270 8252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:48:52.119375 8252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "9c292c19bceb425ba2a232bbc4978ca0"
format_stamp: "Formatted at 2026-09-28 04:48:52 on dist-test-slave-jccs"
server_key: "b62f09811d2b3799e39f6ec4850ae016"
server_key_iv: "c2f7e27ffe3d3875569494430fd00c7c"
server_key_version: "encryptionkey@0"
I20260928 04:48:52.119467 8252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:52.158257 8252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:52.160243 8252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:52.160415 8252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:52.160918 8252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:52.160988 8252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:52.161037 8252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:52.161068 8252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 04:48:52 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570932, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:52.168273 8252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43537
I20260928 04:48:52.168355 8382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43537 every 8 connection(s)
I20260928 04:48:52.168684 8252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:48:52.169289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 8252
I20260928 04:48:52.169413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:52.169638 392 external_mini_cluster.cc:1428] Setting key 9c0523ab37011db3c9b544eeaf20ca3c
I20260928 04:48:52.171128 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.161461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46591 (local address 127.0.98.62:39465)
0928 04:48:52.161526 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.161529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.161770 (+ 241us) server_negotiation.cc:407] Connection header received
0928 04:48:52.162490 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.162492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.162505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.162573 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:52.163472 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.163977 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.165058 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.165346 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.165422 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.168025 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:52.168027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:52.168027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:52.168034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:52.169768 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:52.170163 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:52.170165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:52.170165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:52.170242 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:52.170459 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:52.170460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:52.170461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:52.170597 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:52.170681 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.170953 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.171043 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:48:52.171293 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.161433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:46591)
0928 04:48:52.161609 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.161664 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.162381 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.162650 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.162655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.163064 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:52.163367 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.163374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.164079 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.164080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.164959 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.164961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.165427 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.165429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.165556 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.166091 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:52.166099 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:52.167919 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:52.169859 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:52.169865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:52.169867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:52.170074 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:52.170311 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:52.170312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:52.170312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:52.170408 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:52.170675 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:52.170680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:52.170818 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.170931 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.171078 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260928 04:48:52.171762 8383 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39465
I20260928 04:48:52.171842 8383 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:52.172021 8383 heartbeater.cc:511] Master 127.0.98.62:39465 requested a full tablet report, sending...
I20260928 04:48:52.172456 7899 ts_manager.cc:194] Registered new tserver with Master: 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537)
I20260928 04:48:52.172894 7899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46591
I20260928 04:48:52.177980 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.172768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40827 (local address 127.0.98.62:39465)
0928 04:48:52.173009 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.173011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.173019 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:52.173037 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.173038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.173047 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.173098 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:52.173459 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.173927 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.174617 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.174822 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.174902 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.175656 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:52.175666 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:52.175666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:52.175672 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:52.176872 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:52.177236 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:52.177237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:52.177237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:52.177290 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:52.177490 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:52.177491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:52.177491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:52.177603 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:52.177677 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.177825 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.177893 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260928 04:48:52.177996 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.172712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.98.3:40827)
0928 04:48:52.172819 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.172828 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.172877 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.173110 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.173111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.173316 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:52.173370 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.173371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.174014 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.174015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.174536 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.174537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.174904 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.174905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.174971 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.175321 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:52.175324 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:52.175571 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:52.176960 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:52.176961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:52.176962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:52.177121 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:52.177367 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:52.177367 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:52.177368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:52.177428 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:52.177676 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:52.177677 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:52.177718 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.177801 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.177894 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":64,"threads_started":1}
I20260928 04:48:52.181200 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 04:48:52 dist-test-slave-jccs krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 04:48:52 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:52 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 04:48:52.201803 7968 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:52.196229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.0.98.62:39465)
0928 04:48:52.196289 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.196291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.196392 (+ 101us) server_negotiation.cc:407] Connection header received
0928 04:48:52.196462 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.196463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.196474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.196553 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:52.197387 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.197895 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.198598 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.198769 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.198818 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.201711 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50874: BlockingRecv error: recv got EOF from 127.0.0.1:50874 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 04:48:52 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:52.208503 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.202514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.0.98.62:39465)
0928 04:48:52.202568 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.202570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.202582 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:48:52.202650 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.202650 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.202658 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.202706 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:52.203117 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.203558 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.204165 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.204347 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.204392 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.206436 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:52.206437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:52.206437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:52.206443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:52.207580 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:52.207891 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:52.207891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:52.207892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:52.207935 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:52.208099 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:52.208100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:52.208100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:52.208188 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:52.208267 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.208398 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.208448 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:48:52.210072 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:52.211560 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:52.218091 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.213810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.0.1:54676)
0928 04:48:52.214123 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.214131 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.214182 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.214770 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.214770 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.214772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:52.214806 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.214807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.215772 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.215773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.216967 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.216969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.217801 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.217802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.217876 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.217889 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.217959 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.218012 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":72,"threads_started":1}
I20260928 04:48:52.218107 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.214140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.0.98.1:39317)
0928 04:48:52.214478 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.214482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.214513 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:48:52.214558 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.214560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.214593 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.214702 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:52.214882 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.215687 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.217055 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.217710 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.217790 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.217865 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.217974 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.218012 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":69,"threads_started":1}
I20260928 04:48:52.218405 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.213807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42153 (local address 127.0.0.1:39890)
0928 04:48:52.213929 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.213965 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.214022 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.214681 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.214684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.214687 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:52.214747 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.214748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.216073 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.216074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.217108 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.217110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.218028 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.218029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.218096 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.218101 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.218133 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.218343 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":8,"thread_start_us":53,"threads_started":1}
I20260928 04:48:52.218770 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.214317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.0.98.2:42153)
0928 04:48:52.214464 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.214468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.214479 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:48:52.214517 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.214519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.214534 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.214734 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:52.214804 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.215981 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.217263 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.217929 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.218516 (+ 587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.218588 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.218614 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.218649 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260928 04:48:52.218838 8037 tablet_service.cc:1467] Processing CreateTablet for tablet e6c974c9429f44d4999c150eb742c6de (DEFAULT_TABLE table=test-table [id=0ed957daa07949dc872bb4bbae2bcd30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:52.219149 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.214090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.0.98.3:43537)
0928 04:48:52.214241 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.214245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.214976 (+ 731us) server_negotiation.cc:407] Connection header received
0928 04:48:52.215024 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.215026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.215041 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.215120 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:52.215224 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.216208 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.217449 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.218126 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.218370 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.218969 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.219000 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.219045 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260928 04:48:52.219270 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c974c9429f44d4999c150eb742c6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:52.219297 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.214006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43537 (local address 127.0.0.1:36956)
0928 04:48:52.214894 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.214903 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.214952 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.215128 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.215129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.215131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:52.215167 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.215168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.216287 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.216288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.217364 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.217367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.218624 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.218625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.218678 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.218681 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.218726 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.218762 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"thread_start_us":60,"threads_started":1}
I20260928 04:48:52.219404 8177 tablet_service.cc:1467] Processing CreateTablet for tablet e6c974c9429f44d4999c150eb742c6de (DEFAULT_TABLE table=test-table [id=0ed957daa07949dc872bb4bbae2bcd30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:52.219727 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c974c9429f44d4999c150eb742c6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:52.221733 8317 tablet_service.cc:1467] Processing CreateTablet for tablet e6c974c9429f44d4999c150eb742c6de (DEFAULT_TABLE table=test-table [id=0ed957daa07949dc872bb4bbae2bcd30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:52.222029 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c974c9429f44d4999c150eb742c6de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:52.223505 8412 tablet_bootstrap.cc:492] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: Bootstrap starting.
I20260928 04:48:52.224133 8412 tablet_bootstrap.cc:654] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:52.224449 8412 log.cc:824] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:52.224781 8411 tablet_bootstrap.cc:492] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: Bootstrap starting.
I20260928 04:48:52.225323 8414 tablet_bootstrap.cc:492] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: Bootstrap starting.
I20260928 04:48:52.225416 8411 tablet_bootstrap.cc:654] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:52.225741 8411 log.cc:824] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:52.225929 8414 tablet_bootstrap.cc:654] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:52.226318 8412 tablet_bootstrap.cc:492] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: No bootstrap required, opened a new log
I20260928 04:48:52.226317 8414 log.cc:824] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:52.226401 8412 ts_tablet_manager.cc:1397] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 04:48:52.226532 8411 tablet_bootstrap.cc:492] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: No bootstrap required, opened a new log
I20260928 04:48:52.226601 8411 ts_tablet_manager.cc:1397] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 04:48:52.227164 8414 tablet_bootstrap.cc:492] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: No bootstrap required, opened a new log
I20260928 04:48:52.227231 8414 ts_tablet_manager.cc:1397] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:48:52.228137 8412 raft_consensus.cc:348] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.228281 8412 raft_consensus.cc:374] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:52.228322 8412 raft_consensus.cc:729] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600ab3a7dceb44ac877658947d3344d7, State: Initialized, Role: FOLLOWER
I20260928 04:48:52.228440 8412 consensus_queue.cc:260] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.228608 8411 raft_consensus.cc:348] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.228708 8243 heartbeater.cc:503] Master 127.0.98.62:39465 was elected leader, sending a full tablet report...
I20260928 04:48:52.228704 8412 ts_tablet_manager.cc:1428] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:48:52.228756 8411 raft_consensus.cc:374] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:52.228788 8411 raft_consensus.cc:729] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf275ca2b6f453f9f116f06ff6431bc, State: Initialized, Role: FOLLOWER
I20260928 04:48:52.228892 8411 consensus_queue.cc:260] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.229125 8411 ts_tablet_manager.cc:1428] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:48:52.229231 8103 heartbeater.cc:503] Master 127.0.98.62:39465 was elected leader, sending a full tablet report...
I20260928 04:48:52.229144 8414 raft_consensus.cc:348] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.229307 8414 raft_consensus.cc:374] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:52.229346 8414 raft_consensus.cc:729] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c292c19bceb425ba2a232bbc4978ca0, State: Initialized, Role: FOLLOWER
I20260928 04:48:52.229446 8414 consensus_queue.cc:260] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.229672 8414 ts_tablet_manager.cc:1428] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 04:48:52.229853 8383 heartbeater.cc:503] Master 127.0.98.62:39465 was elected leader, sending a full tablet report...
W20260928 04:48:52.241995 8244 tablet.cc:2369] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:48:52.346210 8104 tablet.cc:2369] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:48:52.347977 8419 raft_consensus.cc:482] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:52.348093 8419 raft_consensus.cc:504] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.348461 8419 leader_election.cc:288] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317), 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2:42153)
I20260928 04:48:52.352356 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.348754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52887 (local address 127.0.98.2:42153)
0928 04:48:52.348795 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.348797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.348807 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:52.348863 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.348863 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.348875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.348923 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:52.349154 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.349918 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.351461 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.352000 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.352074 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.352084 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.352256 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.352292 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:48:52.352356 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.348710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48841 (local address 127.0.98.1:39317)
0928 04:48:52.348754 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.348756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.348884 (+ 128us) server_negotiation.cc:407] Connection header received
0928 04:48:52.349037 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.349043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.349053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.349093 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:52.349266 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.349939 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.351457 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.351999 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.352074 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.352084 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.352252 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.352289 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:48:52.352371 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.348656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42153 (local address 127.0.98.3:52887)
0928 04:48:52.348722 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.348735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.348786 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.348955 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.348956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.348958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:52.349050 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.349052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.350012 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.350013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.351365 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.351368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.352091 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.352092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.352171 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.352177 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.352236 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.352280 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":100}
I20260928 04:48:52.352387 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.348668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.98.3:48841)
0928 04:48:52.348817 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.348829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.348954 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.349128 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.349129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.349130 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:52.349222 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.349223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.350012 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.350013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.351365 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.351368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.352095 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.352096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.352171 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.352178 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.352236 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.352280 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":70,"thread_start_us":98,"threads_started":1}
I20260928 04:48:52.352828 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c974c9429f44d4999c150eb742c6de" candidate_uuid: "9c292c19bceb425ba2a232bbc4978ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600ab3a7dceb44ac877658947d3344d7" is_pre_election: true
I20260928 04:48:52.352973 8197 raft_consensus.cc:2454] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c292c19bceb425ba2a232bbc4978ca0 in term 0.
I20260928 04:48:52.353235 8270 leader_election.cc:302] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 600ab3a7dceb44ac877658947d3344d7, 9c292c19bceb425ba2a232bbc4978ca0; no voters:
I20260928 04:48:52.353315 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c974c9429f44d4999c150eb742c6de" candidate_uuid: "9c292c19bceb425ba2a232bbc4978ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" is_pre_election: true
I20260928 04:48:52.353361 8419 raft_consensus.cc:2781] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:52.353400 8419 raft_consensus.cc:482] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:52.353422 8419 raft_consensus.cc:3037] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:52.353447 8057 raft_consensus.cc:2454] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c292c19bceb425ba2a232bbc4978ca0 in term 0.
I20260928 04:48:52.354353 8419 raft_consensus.cc:504] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.354482 8419 leader_election.cc:288] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [CANDIDATE]: Term 1 election: Requested vote from peers 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317), 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2:42153)
I20260928 04:48:52.354614 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c974c9429f44d4999c150eb742c6de" candidate_uuid: "9c292c19bceb425ba2a232bbc4978ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf275ca2b6f453f9f116f06ff6431bc"
I20260928 04:48:52.354707 8057 raft_consensus.cc:3037] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:52.354722 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c974c9429f44d4999c150eb742c6de" candidate_uuid: "9c292c19bceb425ba2a232bbc4978ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600ab3a7dceb44ac877658947d3344d7"
I20260928 04:48:52.354779 8197 raft_consensus.cc:3037] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:52.355525 8057 raft_consensus.cc:2454] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c292c19bceb425ba2a232bbc4978ca0 in term 1.
I20260928 04:48:52.355525 8197 raft_consensus.cc:2454] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c292c19bceb425ba2a232bbc4978ca0 in term 1.
I20260928 04:48:52.355695 8271 leader_election.cc:302] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf275ca2b6f453f9f116f06ff6431bc, 9c292c19bceb425ba2a232bbc4978ca0; no voters:
I20260928 04:48:52.355785 8419 raft_consensus.cc:2781] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:52.355911 8419 raft_consensus.cc:686] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 LEADER]: Becoming Leader. State: Replica: 9c292c19bceb425ba2a232bbc4978ca0, State: Running, Role: LEADER
I20260928 04:48:52.356002 8419 consensus_queue.cc:237] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.356825 7900 catalog_manager.cc:5495] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c292c19bceb425ba2a232bbc4978ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } health_report { overall_health: HEALTHY } } }
I20260928 04:48:52.363409 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.361082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.0.98.3:43537)
0928 04:48:52.361126 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.361127 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.361136 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:52.361235 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.361236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.361252 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.361329 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:52.361466 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.361927 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.362648 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.362862 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.362931 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.363106 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:52.363201 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:52.363291 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.363319 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.363336 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:48:52.364854 8197 raft_consensus.cc:1261] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Refusing update from remote peer 9c292c19bceb425ba2a232bbc4978ca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:52.364878 8057 raft_consensus.cc:1261] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Refusing update from remote peer 9c292c19bceb425ba2a232bbc4978ca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:52.365154 8419 consensus_queue.cc:1035] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:52.365311 8421 consensus_queue.cc:1035] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:52.367650 8425 mvcc.cc:205] Tried to move back new op lower bound from 7334178538964021248 to 7334178538930638848. Current Snapshot: MvccSnapshot[applied={T|T < 7334178538964021248}]
I20260928 04:48:52.370025 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:52.370111 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:52.370908 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626: Sending DeleteTablet for 3 replicas of tablet e6c974c9429f44d4999c150eb742c6de
I20260928 04:48:52.371271 8037 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c974c9429f44d4999c150eb742c6de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54676
I20260928 04:48:52.371264 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c974c9429f44d4999c150eb742c6de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36956
I20260928 04:48:52.371340 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c974c9429f44d4999c150eb742c6de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39890
I20260928 04:48:52.371534 8434 tablet_replica.cc:330] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: stopping tablet replica
I20260928 04:48:52.371706 8434 raft_consensus.cc:2229] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:52.371806 8436 tablet_replica.cc:330] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: stopping tablet replica
I20260928 04:48:52.371804 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 04:48:52.371894 8436 raft_consensus.cc:2229] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 04:48:52.371954 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:52.372002 8436 raft_consensus.cc:2258] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:52.372418 8436 ts_tablet_manager.cc:1905] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:52.372363 8434 raft_consensus.cc:2258] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:52.372977 8435 tablet_replica.cc:330] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: stopping tablet replica
I20260928 04:48:52.373116 8435 raft_consensus.cc:2229] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:52.373294 8435 raft_consensus.cc:2258] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:52.373400 8037 tablet_service.cc:1467] Processing CreateTablet for tablet a9fbe39e410341d6840df9d020090ebb (DEFAULT_TABLE table=test-table [id=ec4fb810bb244419a2ba606e8b51155b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:52.373512 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fbe39e410341d6840df9d020090ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:52.373874 8435 ts_tablet_manager.cc:1905] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:52.374126 8436 ts_tablet_manager.cc:1918] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:52.374198 8436 log.cc:1196] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/e6c974c9429f44d4999c150eb742c6de
I20260928 04:48:52.374452 8436 ts_tablet_manager.cc:1939] T e6c974c9429f44d4999c150eb742c6de P 600ab3a7dceb44ac877658947d3344d7: Deleting consensus metadata
I20260928 04:48:52.374843 8317 tablet_service.cc:1467] Processing CreateTablet for tablet a9fbe39e410341d6840df9d020090ebb (DEFAULT_TABLE table=test-table [id=ec4fb810bb244419a2ba606e8b51155b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:52.374946 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fbe39e410341d6840df9d020090ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:52.375622 8435 ts_tablet_manager.cc:1918] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:52.375705 8435 log.cc:1196] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/e6c974c9429f44d4999c150eb742c6de
I20260928 04:48:52.375928 8435 ts_tablet_manager.cc:1939] T e6c974c9429f44d4999c150eb742c6de P 9c292c19bceb425ba2a232bbc4978ca0: Deleting consensus metadata
I20260928 04:48:52.376144 8414 tablet_bootstrap.cc:492] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: Bootstrap starting.
I20260928 04:48:52.376171 8411 tablet_bootstrap.cc:492] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc: Bootstrap starting.
I20260928 04:48:52.376484 8414 tablet_bootstrap.cc:654] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:52.376529 8411 tablet_bootstrap.cc:654] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:52.376991 8414 tablet_bootstrap.cc:492] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: No bootstrap required, opened a new log
I20260928 04:48:52.377034 7886 catalog_manager.cc:4841] TS 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2:42153): tablet e6c974c9429f44d4999c150eb742c6de (table test-table [id=0ed957daa07949dc872bb4bbae2bcd30]) successfully deleted
I20260928 04:48:52.377054 8414 ts_tablet_manager.cc:1397] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 04:48:52.377215 8414 raft_consensus.cc:348] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.377236 8177 tablet_service.cc:1467] Processing CreateTablet for tablet a9fbe39e410341d6840df9d020090ebb (DEFAULT_TABLE table=test-table [id=ec4fb810bb244419a2ba606e8b51155b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:52.377286 8414 raft_consensus.cc:374] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:52.377280 7887 catalog_manager.cc:4841] TS 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537): tablet e6c974c9429f44d4999c150eb742c6de (table test-table [id=0ed957daa07949dc872bb4bbae2bcd30]) successfully deleted
I20260928 04:48:52.377310 8414 raft_consensus.cc:729] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c292c19bceb425ba2a232bbc4978ca0, State: Initialized, Role: FOLLOWER
I20260928 04:48:52.377329 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fbe39e410341d6840df9d020090ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:52.377351 8414 consensus_queue.cc:260] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.377442 8414 ts_tablet_manager.cc:1428] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:52.377954 8434 ts_tablet_manager.cc:1905] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:52.378396 8412 tablet_bootstrap.cc:492] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7: Bootstrap starting.
I20260928 04:48:52.378530 8411 tablet_bootstrap.cc:492] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc: No bootstrap required, opened a new log
I20260928 04:48:52.378582 8411 ts_tablet_manager.cc:1397] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:48:52.378724 8411 raft_consensus.cc:348] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.378794 8411 raft_consensus.cc:374] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:52.378814 8411 raft_consensus.cc:729] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf275ca2b6f453f9f116f06ff6431bc, State: Initialized, Role: FOLLOWER
I20260928 04:48:52.378788 8412 tablet_bootstrap.cc:654] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:52.378868 8411 consensus_queue.cc:260] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.378994 8411 ts_tablet_manager.cc:1428] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:52.379637 8434 ts_tablet_manager.cc:1918] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:52.379706 8434 log.cc:1196] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/e6c974c9429f44d4999c150eb742c6de
I20260928 04:48:52.380208 8434 ts_tablet_manager.cc:1939] T e6c974c9429f44d4999c150eb742c6de P 1bf275ca2b6f453f9f116f06ff6431bc: Deleting consensus metadata
I20260928 04:48:52.380566 7887 catalog_manager.cc:4841] TS 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317): tablet e6c974c9429f44d4999c150eb742c6de (table test-table [id=0ed957daa07949dc872bb4bbae2bcd30]) successfully deleted
I20260928 04:48:52.381562 8412 tablet_bootstrap.cc:492] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7: No bootstrap required, opened a new log
I20260928 04:48:52.381606 8412 ts_tablet_manager.cc:1397] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 04:48:52.381762 8412 raft_consensus.cc:348] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.381811 8412 raft_consensus.cc:374] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:52.381825 8412 raft_consensus.cc:729] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600ab3a7dceb44ac877658947d3344d7, State: Initialized, Role: FOLLOWER
I20260928 04:48:52.381876 8412 consensus_queue.cc:260] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.381969 8412 ts_tablet_manager.cc:1428] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260928 04:48:52.419293 8384 tablet.cc:2369] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:48:52.531214 8429 raft_consensus.cc:482] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:52.531309 8429 raft_consensus.cc:504] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.531482 8429 leader_election.cc:288] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317), 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2:42153)
I20260928 04:48:52.531723 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fbe39e410341d6840df9d020090ebb" candidate_uuid: "9c292c19bceb425ba2a232bbc4978ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" is_pre_election: true
I20260928 04:48:52.531807 8057 raft_consensus.cc:2454] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c292c19bceb425ba2a232bbc4978ca0 in term 0.
I20260928 04:48:52.531756 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fbe39e410341d6840df9d020090ebb" candidate_uuid: "9c292c19bceb425ba2a232bbc4978ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600ab3a7dceb44ac877658947d3344d7" is_pre_election: true
I20260928 04:48:52.531889 8197 raft_consensus.cc:2454] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c292c19bceb425ba2a232bbc4978ca0 in term 0.
I20260928 04:48:52.531989 8271 leader_election.cc:302] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf275ca2b6f453f9f116f06ff6431bc, 9c292c19bceb425ba2a232bbc4978ca0; no voters:
I20260928 04:48:52.532094 8429 raft_consensus.cc:2781] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:52.532136 8429 raft_consensus.cc:482] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:52.532151 8429 raft_consensus.cc:3037] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:52.532819 8429 raft_consensus.cc:504] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.532987 8429 leader_election.cc:288] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [CANDIDATE]: Term 1 election: Requested vote from peers 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317), 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2:42153)
I20260928 04:48:52.533138 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fbe39e410341d6840df9d020090ebb" candidate_uuid: "9c292c19bceb425ba2a232bbc4978ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600ab3a7dceb44ac877658947d3344d7"
I20260928 04:48:52.533165 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fbe39e410341d6840df9d020090ebb" candidate_uuid: "9c292c19bceb425ba2a232bbc4978ca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf275ca2b6f453f9f116f06ff6431bc"
I20260928 04:48:52.533254 8197 raft_consensus.cc:3037] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:52.533265 8057 raft_consensus.cc:3037] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:52.533849 8197 raft_consensus.cc:2454] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c292c19bceb425ba2a232bbc4978ca0 in term 1.
I20260928 04:48:52.533982 8057 raft_consensus.cc:2454] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c292c19bceb425ba2a232bbc4978ca0 in term 1.
I20260928 04:48:52.534080 8270 leader_election.cc:302] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 600ab3a7dceb44ac877658947d3344d7, 9c292c19bceb425ba2a232bbc4978ca0; no voters:
I20260928 04:48:52.534245 8429 raft_consensus.cc:2781] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:52.534300 8429 raft_consensus.cc:686] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 LEADER]: Becoming Leader. State: Replica: 9c292c19bceb425ba2a232bbc4978ca0, State: Running, Role: LEADER
I20260928 04:48:52.534332 8429 consensus_queue.cc:237] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.534946 7899 catalog_manager.cc:5495] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c292c19bceb425ba2a232bbc4978ca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } health_report { overall_health: HEALTHY } } }
Sep 28 04:48:52 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:52.592185 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.585981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.0.98.62:39465)
0928 04:48:52.586059 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.586061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.586140 (+ 79us) server_negotiation.cc:407] Connection header received
0928 04:48:52.586185 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.586186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.586195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.586247 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:52.586673 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.587139 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.587775 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.587938 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.587986 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.590003 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:52.590004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:52.590004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:52.590009 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:52.591150 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:52.591520 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:52.591521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:52.591521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:52.591565 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:52.591727 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:52.591727 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:52.591727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:52.591817 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:52.591864 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.591990 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.592087 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":51}
I20260928 04:48:52.592186 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.585917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39465 (local address 127.0.0.1:50892)
0928 04:48:52.586063 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.586072 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.586142 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.586263 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.586264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.586555 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:52.586603 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.586615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.587211 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.587212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.587712 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.587713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.587997 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.587998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.588033 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.588577 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:52.588581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:52.589933 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:52.591246 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:52.591252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:52.591254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:52.591444 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:52.591620 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:52.591621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:52.591621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:52.591676 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:52.591867 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:52.591872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:52.591911 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.591959 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.592046 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260928 04:48:52.594214 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50892:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 04:48:52.594431 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:52.595916 8177 tablet_service.cc:1467] Processing CreateTablet for tablet 997396ac740845af9075754de1d1545a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0de5f9a5f9394866b837d472d55abc3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:48:52.595930 8037 tablet_service.cc:1467] Processing CreateTablet for tablet 997396ac740845af9075754de1d1545a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0de5f9a5f9394866b837d472d55abc3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:48:52.596082 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997396ac740845af9075754de1d1545a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:52.596091 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997396ac740845af9075754de1d1545a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:52.597210 8412 tablet_bootstrap.cc:492] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: Bootstrap starting.
I20260928 04:48:52.597363 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 997396ac740845af9075754de1d1545a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0de5f9a5f9394866b837d472d55abc3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 04:48:52.597484 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997396ac740845af9075754de1d1545a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:52.597586 8412 tablet_bootstrap.cc:654] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:52.598141 8412 tablet_bootstrap.cc:492] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: No bootstrap required, opened a new log
I20260928 04:48:52.598186 8412 ts_tablet_manager.cc:1397] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:52.598331 8412 raft_consensus.cc:348] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.598388 8412 raft_consensus.cc:374] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:52.598402 8412 raft_consensus.cc:729] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600ab3a7dceb44ac877658947d3344d7, State: Initialized, Role: FOLLOWER
I20260928 04:48:52.598454 8412 consensus_queue.cc:260] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.598572 8412 ts_tablet_manager.cc:1428] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:52.598838 8414 tablet_bootstrap.cc:492] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: Bootstrap starting.
I20260928 04:48:52.598874 8411 tablet_bootstrap.cc:492] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: Bootstrap starting.
I20260928 04:48:52.599171 8414 tablet_bootstrap.cc:654] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:52.599269 8411 tablet_bootstrap.cc:654] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:52.599644 8417 tablet_replica.cc:439] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } } }
I20260928 04:48:52.599752 8417 tablet_replica.cc:442] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:52.599771 8414 tablet_bootstrap.cc:492] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: No bootstrap required, opened a new log
I20260928 04:48:52.599817 8414 ts_tablet_manager.cc:1397] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:52.599941 8414 raft_consensus.cc:348] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.600020 8414 raft_consensus.cc:374] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:52.600024 8411 tablet_bootstrap.cc:492] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: No bootstrap required, opened a new log
I20260928 04:48:52.600044 8414 raft_consensus.cc:729] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c292c19bceb425ba2a232bbc4978ca0, State: Initialized, Role: FOLLOWER
I20260928 04:48:52.600095 8411 ts_tablet_manager.cc:1397] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:52.600100 8414 consensus_queue.cc:260] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.600196 8414 ts_tablet_manager.cc:1428] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:52.600234 8411 raft_consensus.cc:348] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.600304 8411 raft_consensus.cc:374] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:52.600338 8411 raft_consensus.cc:729] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf275ca2b6f453f9f116f06ff6431bc, State: Initialized, Role: FOLLOWER
I20260928 04:48:52.600373 8411 consensus_queue.cc:260] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.600471 8411 ts_tablet_manager.cc:1428] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:52.600206 8429 tablet_replica.cc:439] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } } }
I20260928 04:48:52.600512 8429 tablet_replica.cc:442] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:52.600535 8418 tablet_replica.cc:439] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } } }
I20260928 04:48:52.600623 8418 tablet_replica.cc:442] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:52.899569 8417 raft_consensus.cc:482] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:52.899669 8417 raft_consensus.cc:504] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.900032 8417 leader_election.cc:288] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317), 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537)
I20260928 04:48:52.902953 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.900254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.98.2:46307)
0928 04:48:52.900394 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.900408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.900480 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.900636 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.900637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.900640 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:52.900698 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.900699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.901368 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.901368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.902167 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.902169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.902705 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.902706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.902772 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.902776 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.902842 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.902882 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260928 04:48:52.902956 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.900356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46307 (local address 127.0.98.1:39317)
0928 04:48:52.900495 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.900497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.900505 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:52.900534 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.900535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.900546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.900621 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:52.900768 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.901287 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.902267 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.902628 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.902707 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.902754 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.902862 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.902894 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260928 04:48:52.903153 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997396ac740845af9075754de1d1545a" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" is_pre_election: true
I20260928 04:48:52.903205 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.900441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43537 (local address 127.0.98.2:51859)
0928 04:48:52.900535 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.900545 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.900596 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.900766 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.900766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.900768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:48:52.900832 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.900834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.901535 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.901536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.902366 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.902368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.902933 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.902934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.903027 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.903031 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.903085 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.903130 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":32,"threads_started":1}
I20260928 04:48:52.903240 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.900505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51859 (local address 127.0.98.3:43537)
0928 04:48:52.900600 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.900601 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.900610 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:52.900681 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.900682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.900690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.900763 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:48:52.900911 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.901458 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.902445 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.902858 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.902933 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.902944 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.903132 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.903176 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":45,"threads_started":1}
I20260928 04:48:52.903231 8057 raft_consensus.cc:2454] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 0.
I20260928 04:48:52.903426 8131 leader_election.cc:302] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf275ca2b6f453f9f116f06ff6431bc, 600ab3a7dceb44ac877658947d3344d7; no voters:
I20260928 04:48:52.903407 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997396ac740845af9075754de1d1545a" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c292c19bceb425ba2a232bbc4978ca0" is_pre_election: true
I20260928 04:48:52.903549 8337 raft_consensus.cc:2454] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 0.
I20260928 04:48:52.903561 8417 raft_consensus.cc:2781] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:52.903616 8417 raft_consensus.cc:482] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:52.903649 8417 raft_consensus.cc:3037] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:52.904284 8417 raft_consensus.cc:504] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.904371 8417 leader_election.cc:288] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317), 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537)
I20260928 04:48:52.904534 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997396ac740845af9075754de1d1545a" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf275ca2b6f453f9f116f06ff6431bc"
I20260928 04:48:52.904539 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997396ac740845af9075754de1d1545a" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c292c19bceb425ba2a232bbc4978ca0"
I20260928 04:48:52.904593 8057 raft_consensus.cc:3037] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:52.904615 8337 raft_consensus.cc:3037] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:52.905229 8057 raft_consensus.cc:2454] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 1.
I20260928 04:48:52.905390 8131 leader_election.cc:302] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf275ca2b6f453f9f116f06ff6431bc, 600ab3a7dceb44ac877658947d3344d7; no voters:
I20260928 04:48:52.905445 8337 raft_consensus.cc:2454] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 1.
I20260928 04:48:52.905499 8417 raft_consensus.cc:2781] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:52.905601 8417 raft_consensus.cc:686] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [term 1 LEADER]: Becoming Leader. State: Replica: 600ab3a7dceb44ac877658947d3344d7, State: Running, Role: LEADER
I20260928 04:48:52.905699 8417 consensus_queue.cc:237] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:52.905983 8454 tablet_replica.cc:439] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: TxnStatusTablet state changed. Reason: New leader 600ab3a7dceb44ac877658947d3344d7. Latest consensus state: current_term: 1 leader_uuid: "600ab3a7dceb44ac877658947d3344d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } } }
I20260928 04:48:52.906148 8454 tablet_replica.cc:442] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:48:52.906325 8456 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 04:48:52.906435 8456 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 04:48:52.906344 7899 catalog_manager.cc:5495] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "600ab3a7dceb44ac877658947d3344d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } health_report { overall_health: UNKNOWN } } }
I20260928 04:48:52.953137 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.950529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.0.98.2:42153)
0928 04:48:52.950669 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.950671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.950679 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:52.950721 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.950722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.950738 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.950810 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:52.950963 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.951381 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.952439 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.952609 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.952656 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.952849 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:52.952939 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:52.952988 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.953062 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.953079 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260928 04:48:52.953217 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.950427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42153 (local address 127.0.0.1:39904)
0928 04:48:52.950510 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.950524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.950573 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.950835 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.950836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.950837 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:48:52.950889 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.950891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.951446 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.951447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.952356 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.952358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.952682 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.952683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.952749 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.952765 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:48:52.952998 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:48:52.953005 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.953057 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.953106 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 04:48:52.954540 8057 raft_consensus.cc:1261] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Refusing update from remote peer 600ab3a7dceb44ac877658947d3344d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:52.954643 8337 raft_consensus.cc:1261] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Refusing update from remote peer 600ab3a7dceb44ac877658947d3344d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:52.954829 8454 consensus_queue.cc:1035] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:52.954939 8417 consensus_queue.cc:1035] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:52.956072 8429 tablet_replica.cc:439] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: TxnStatusTablet state changed. Reason: New leader 600ab3a7dceb44ac877658947d3344d7. Latest consensus state: current_term: 1 leader_uuid: "600ab3a7dceb44ac877658947d3344d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } } }
I20260928 04:48:52.956163 8429 tablet_replica.cc:442] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:52.956465 8418 tablet_replica.cc:439] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: TxnStatusTablet state changed. Reason: New leader 600ab3a7dceb44ac877658947d3344d7. Latest consensus state: current_term: 1 leader_uuid: "600ab3a7dceb44ac877658947d3344d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } } }
I20260928 04:48:52.956568 8418 tablet_replica.cc:442] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:52.956549 8460 mvcc.cc:205] Tried to move back new op lower bound from 7334178541379457024 to 7334178541182083072. Current Snapshot: MvccSnapshot[applied={T|T < 7334178541379457024}]
I20260928 04:48:52.956619 8454 tablet_replica.cc:439] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "600ab3a7dceb44ac877658947d3344d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } } }
I20260928 04:48:52.956701 8454 tablet_replica.cc:442] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:48:52.956715 8465 tablet_replica.cc:439] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "600ab3a7dceb44ac877658947d3344d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } } }
I20260928 04:48:52.956815 8465 tablet_replica.cc:442] T 997396ac740845af9075754de1d1545a P 600ab3a7dceb44ac877658947d3344d7: This TxnStatusTablet replica's current role is: LEADER
I20260928 04:48:52.956858 8429 tablet_replica.cc:439] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "600ab3a7dceb44ac877658947d3344d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } } }
I20260928 04:48:52.956926 8429 tablet_replica.cc:442] T 997396ac740845af9075754de1d1545a P 9c292c19bceb425ba2a232bbc4978ca0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:52.957019 8462 mvcc.cc:205] Tried to move back new op lower bound from 7334178541379457024 to 7334178541182083072. Current Snapshot: MvccSnapshot[applied={T|T < 7334178541379457024}]
I20260928 04:48:52.957479 8418 tablet_replica.cc:439] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "600ab3a7dceb44ac877658947d3344d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } } }
I20260928 04:48:52.957567 8418 tablet_replica.cc:442] T 997396ac740845af9075754de1d1545a P 1bf275ca2b6f453f9f116f06ff6431bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 04:48:52.963255 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.960785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48831 (local address 127.0.98.2:42153)
0928 04:48:52.960826 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.960829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.960943 (+ 114us) server_negotiation.cc:407] Connection header received
0928 04:48:52.960966 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.960967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.960976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.961041 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:52.961118 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.961769 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.962519 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.962767 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.962839 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.962955 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:52.962986 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:52.963048 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.963173 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.963194 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:48:52.963290 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.960723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42153 (local address 127.0.98.3:48831)
0928 04:48:52.960864 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.960874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.960931 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.961041 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.961042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.961044 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:48:52.961078 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.961079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.961840 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.961840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.962439 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.962441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.962841 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.962842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.962877 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.962888 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:48:52.963064 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:48:52.963066 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.963158 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.963208 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260928 04:48:52.963419 8429 consensus_queue.cc:1035] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:52.965250 8421 consensus_queue.cc:1035] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:52.974252 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.971312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43865 (local address 127.0.98.3:43537)
0928 04:48:52.971393 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:52.971395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:52.971523 (+ 128us) server_negotiation.cc:407] Connection header received
0928 04:48:52.971618 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:52.971619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:52.971637 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:52.971679 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:52.971831 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.972488 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.973614 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.973799 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.973849 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.974049 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:52.974071 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:52.974104 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:52.974191 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 04:48:52.974206 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 04:48:52.974633 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:52.971289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43537 (local address 127.0.98.2:43865)
0928 04:48:52.971461 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:52.971473 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:52.971537 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:52.971712 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:52.971713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:52.971715 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:48:52.971771 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.971773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.972559 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.972560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.973530 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:52.973533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:52.973885 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:52.973886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:52.973951 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:52.973966 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:48:52.974125 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:48:52.974126 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:48:52.974187 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:48:52.974228 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260928 04:48:52.982836 7900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:52.982945 7900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:52.983556 7900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626: Sending DeleteTablet for 3 replicas of tablet a9fbe39e410341d6840df9d020090ebb
I20260928 04:48:52.983834 8037 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fbe39e410341d6840df9d020090ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54676
I20260928 04:48:52.983834 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fbe39e410341d6840df9d020090ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36956
I20260928 04:48:52.986703 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fbe39e410341d6840df9d020090ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39890
I20260928 04:48:52.986919 8482 tablet_replica.cc:330] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7: stopping tablet replica
I20260928 04:48:52.986992 8482 raft_consensus.cc:2229] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:52.987035 8482 raft_consensus.cc:2258] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:52.987274 8482 ts_tablet_manager.cc:1905] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:52.988684 8482 ts_tablet_manager.cc:1918] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:52.988761 8482 log.cc:1196] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/a9fbe39e410341d6840df9d020090ebb
I20260928 04:48:52.989053 8482 ts_tablet_manager.cc:1939] T a9fbe39e410341d6840df9d020090ebb P 600ab3a7dceb44ac877658947d3344d7: Deleting consensus metadata
I20260928 04:48:52.989394 7886 catalog_manager.cc:4841] TS 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2:42153): tablet a9fbe39e410341d6840df9d020090ebb (table test-table [id=ec4fb810bb244419a2ba606e8b51155b]) successfully deleted
Sep 28 04:48:52 dist-test-slave-jccs krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:52.995765 8480 tablet_replica.cc:330] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc: stopping tablet replica
I20260928 04:48:52.995831 8480 raft_consensus.cc:2229] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Raft consensus shutting down.
Password for test-admin@KRBTEST.COM: I20260928 04:48:52.995968 8479 tablet_replica.cc:330] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: stopping tablet replica
I20260928 04:48:52.996037 8479 raft_consensus.cc:2229] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:52.996105 8479 raft_consensus.cc:2258] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:52.996174 8480 raft_consensus.cc:2258] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:52.996446 8479 ts_tablet_manager.cc:1905] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:52.996539 8480 ts_tablet_manager.cc:1905] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:52.997918 8479 ts_tablet_manager.cc:1918] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:52.997977 8479 log.cc:1196] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/a9fbe39e410341d6840df9d020090ebb
I20260928 04:48:52.998190 8479 ts_tablet_manager.cc:1939] T a9fbe39e410341d6840df9d020090ebb P 9c292c19bceb425ba2a232bbc4978ca0: Deleting consensus metadata
I20260928 04:48:52.998361 8480 ts_tablet_manager.cc:1918] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:52.998457 8480 log.cc:1196] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/a9fbe39e410341d6840df9d020090ebb
I20260928 04:48:52.998488 7887 catalog_manager.cc:4841] TS 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537): tablet a9fbe39e410341d6840df9d020090ebb (table test-table [id=ec4fb810bb244419a2ba606e8b51155b]) successfully deleted
I20260928 04:48:52.998762 8480 ts_tablet_manager.cc:1939] T a9fbe39e410341d6840df9d020090ebb P 1bf275ca2b6f453f9f116f06ff6431bc: Deleting consensus metadata
I20260928 04:48:52.999094 7887 catalog_manager.cc:4841] TS 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317): tablet a9fbe39e410341d6840df9d020090ebb (table test-table [id=ec4fb810bb244419a2ba606e8b51155b]) successfully deleted
Sep 28 04:48:53 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:53 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 04:48:53.008037 7968 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:53.002241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.0.98.62:39465)
0928 04:48:53.002287 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:53.002289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:53.002428 (+ 139us) server_negotiation.cc:407] Connection header received
0928 04:48:53.002493 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:53.002494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:53.002503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:53.002549 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:53.003216 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:53.003686 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:53.004786 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:53.004955 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:53.004998 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:53.007955 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50896: BlockingRecv error: recv got EOF from 127.0.0.1:50896 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 04:48:53 dist-test-slave-jccs krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 04:48:53.016005 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:53.008883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.0.98.62:39465)
0928 04:48:53.008919 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:53.008920 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:53.008968 (+ 48us) server_negotiation.cc:407] Connection header received
0928 04:48:53.009021 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:53.009022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:53.009032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:53.009084 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:53.009576 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:53.010283 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:53.010906 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:53.011189 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:53.011260 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:53.013427 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:53.013428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:53.013428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:53.013435 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:53.014911 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:53.015215 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:53.015216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:53.015216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:53.015277 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:53.015450 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:53.015452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:53.015452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:53.015603 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:53.015722 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:53.015804 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 04:48:53.015895 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:48:53.017570 7900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:48:53.017751 7900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:53.019016 8037 tablet_service.cc:1467] Processing CreateTablet for tablet 956ce34b2c3d4372a84b494d0df74a17 (DEFAULT_TABLE table=test-table [id=fbad8274724b40e88dc78ffe0edf88e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:53.019034 8177 tablet_service.cc:1467] Processing CreateTablet for tablet 956ce34b2c3d4372a84b494d0df74a17 (DEFAULT_TABLE table=test-table [id=fbad8274724b40e88dc78ffe0edf88e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:53.019068 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 956ce34b2c3d4372a84b494d0df74a17 (DEFAULT_TABLE table=test-table [id=fbad8274724b40e88dc78ffe0edf88e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:53.019150 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956ce34b2c3d4372a84b494d0df74a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:53.019164 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956ce34b2c3d4372a84b494d0df74a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:53.019161 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956ce34b2c3d4372a84b494d0df74a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:53.020393 8412 tablet_bootstrap.cc:492] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7: Bootstrap starting.
I20260928 04:48:53.020393 8414 tablet_bootstrap.cc:492] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0: Bootstrap starting.
I20260928 04:48:53.020419 8411 tablet_bootstrap.cc:492] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc: Bootstrap starting.
I20260928 04:48:53.020746 8414 tablet_bootstrap.cc:654] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:53.020756 8412 tablet_bootstrap.cc:654] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:53.020860 8411 tablet_bootstrap.cc:654] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:53.021556 8412 tablet_bootstrap.cc:492] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7: No bootstrap required, opened a new log
I20260928 04:48:53.021587 8411 tablet_bootstrap.cc:492] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc: No bootstrap required, opened a new log
I20260928 04:48:53.021606 8412 ts_tablet_manager.cc:1397] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:53.021626 8411 ts_tablet_manager.cc:1397] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:53.021762 8411 raft_consensus.cc:348] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.021766 8412 raft_consensus.cc:348] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.021838 8411 raft_consensus.cc:374] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:53.021840 8412 raft_consensus.cc:374] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:53.021858 8411 raft_consensus.cc:729] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf275ca2b6f453f9f116f06ff6431bc, State: Initialized, Role: FOLLOWER
I20260928 04:48:53.021867 8412 raft_consensus.cc:729] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600ab3a7dceb44ac877658947d3344d7, State: Initialized, Role: FOLLOWER
I20260928 04:48:53.021900 8411 consensus_queue.cc:260] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.021916 8412 consensus_queue.cc:260] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.021978 8411 ts_tablet_manager.cc:1428] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:53.021994 8412 ts_tablet_manager.cc:1428] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:53.022516 8414 tablet_bootstrap.cc:492] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0: No bootstrap required, opened a new log
I20260928 04:48:53.022569 8414 ts_tablet_manager.cc:1397] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:48:53.022703 8414 raft_consensus.cc:348] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.022784 8414 raft_consensus.cc:374] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:53.022811 8414 raft_consensus.cc:729] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c292c19bceb425ba2a232bbc4978ca0, State: Initialized, Role: FOLLOWER
I20260928 04:48:53.022853 8414 consensus_queue.cc:260] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.022939 8414 ts_tablet_manager.cc:1428] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:53.073385 8454 raft_consensus.cc:482] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:53.073484 8454 raft_consensus.cc:504] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.073649 8454 leader_election.cc:288] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317), 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537)
I20260928 04:48:53.073841 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956ce34b2c3d4372a84b494d0df74a17" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" is_pre_election: true
I20260928 04:48:53.073927 8056 raft_consensus.cc:2454] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 0.
I20260928 04:48:53.073889 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956ce34b2c3d4372a84b494d0df74a17" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c292c19bceb425ba2a232bbc4978ca0" is_pre_election: true
I20260928 04:48:53.074007 8337 raft_consensus.cc:2454] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 0.
I20260928 04:48:53.074083 8131 leader_election.cc:302] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf275ca2b6f453f9f116f06ff6431bc, 600ab3a7dceb44ac877658947d3344d7; no voters:
I20260928 04:48:53.074188 8454 raft_consensus.cc:2781] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:53.074242 8454 raft_consensus.cc:482] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:53.074261 8454 raft_consensus.cc:3037] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:53.074944 8454 raft_consensus.cc:504] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.075058 8454 leader_election.cc:288] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317), 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537)
I20260928 04:48:53.075204 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956ce34b2c3d4372a84b494d0df74a17" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c292c19bceb425ba2a232bbc4978ca0"
I20260928 04:48:53.075290 8337 raft_consensus.cc:3037] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:53.075265 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956ce34b2c3d4372a84b494d0df74a17" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf275ca2b6f453f9f116f06ff6431bc"
I20260928 04:48:53.075325 8056 raft_consensus.cc:3037] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:53.076043 8337 raft_consensus.cc:2454] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 1.
I20260928 04:48:53.076247 8131 leader_election.cc:302] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 600ab3a7dceb44ac877658947d3344d7, 9c292c19bceb425ba2a232bbc4978ca0; no voters:
I20260928 04:48:53.076350 8454 raft_consensus.cc:2781] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:53.076404 8454 raft_consensus.cc:686] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 1 LEADER]: Becoming Leader. State: Replica: 600ab3a7dceb44ac877658947d3344d7, State: Running, Role: LEADER
I20260928 04:48:53.076404 8056 raft_consensus.cc:2454] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 1.
I20260928 04:48:53.076454 8454 consensus_queue.cc:237] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.077045 7899 catalog_manager.cc:5495] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "600ab3a7dceb44ac877658947d3344d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } health_report { overall_health: UNKNOWN } } }
I20260928 04:48:53.081549 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:53.079609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.98.2:42153)
0928 04:48:53.079678 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:53.079680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:53.079686 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:48:53.079704 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:53.079705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:53.079713 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:53.079756 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:53.079837 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:53.080306 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:53.080985 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:53.081156 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:53.081262 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:53.081360 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:53.081383 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:53.081415 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:53.081447 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 04:48:53.081461 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:48:53.082314 8337 raft_consensus.cc:1261] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Refusing update from remote peer 600ab3a7dceb44ac877658947d3344d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:53.082355 8056 raft_consensus.cc:1261] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Refusing update from remote peer 600ab3a7dceb44ac877658947d3344d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:53.082559 8465 consensus_queue.cc:1035] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:53.082638 8454 consensus_queue.cc:1035] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:53.085589 7900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:53.085683 7900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:53.086357 7900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626: Sending DeleteTablet for 3 replicas of tablet 956ce34b2c3d4372a84b494d0df74a17
I20260928 04:48:53.086585 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet 956ce34b2c3d4372a84b494d0df74a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39890
I20260928 04:48:53.086615 8037 tablet_service.cc:1514] Processing DeleteTablet for tablet 956ce34b2c3d4372a84b494d0df74a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54676
I20260928 04:48:53.086632 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet 956ce34b2c3d4372a84b494d0df74a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36956
I20260928 04:48:53.086686 8482 tablet_replica.cc:330] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7: stopping tablet replica
I20260928 04:48:53.086704 8479 tablet_replica.cc:330] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0: stopping tablet replica
I20260928 04:48:53.086704 8480 tablet_replica.cc:330] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc: stopping tablet replica
I20260928 04:48:53.086737 8482 raft_consensus.cc:2229] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:53.086743 8479 raft_consensus.cc:2229] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:53.086750 8480 raft_consensus.cc:2229] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:53.086778 8479 raft_consensus.cc:2258] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:53.086784 8480 raft_consensus.cc:2258] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:53.086818 8482 raft_consensus.cc:2258] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:53.087078 7900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 04:48:53.087183 8482 ts_tablet_manager.cc:1905] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:53.087208 8480 ts_tablet_manager.cc:1905] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:48:53.087226 7900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:48:53.088290 8317 tablet_service.cc:1467] Processing CreateTablet for tablet b421cab88412405f8037dfc0dcc2c8a3 (DEFAULT_TABLE table=test-table [id=e57e3ec74f7f4b52a8decec666bbbe81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:53.088387 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b421cab88412405f8037dfc0dcc2c8a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:53.088730 8482 ts_tablet_manager.cc:1918] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:53.088783 8482 log.cc:1196] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/956ce34b2c3d4372a84b494d0df74a17
I20260928 04:48:53.088948 8480 ts_tablet_manager.cc:1918] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:53.088989 8482 ts_tablet_manager.cc:1939] T 956ce34b2c3d4372a84b494d0df74a17 P 600ab3a7dceb44ac877658947d3344d7: Deleting consensus metadata
I20260928 04:48:53.088997 8480 log.cc:1196] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/956ce34b2c3d4372a84b494d0df74a17
I20260928 04:48:53.089274 8480 ts_tablet_manager.cc:1939] T 956ce34b2c3d4372a84b494d0df74a17 P 1bf275ca2b6f453f9f116f06ff6431bc: Deleting consensus metadata
I20260928 04:48:53.089725 8414 tablet_bootstrap.cc:492] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0: Bootstrap starting.
I20260928 04:48:53.089862 8037 tablet_service.cc:1467] Processing CreateTablet for tablet b421cab88412405f8037dfc0dcc2c8a3 (DEFAULT_TABLE table=test-table [id=e57e3ec74f7f4b52a8decec666bbbe81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:53.089955 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b421cab88412405f8037dfc0dcc2c8a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:53.090018 8414 tablet_bootstrap.cc:654] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:53.090564 8414 tablet_bootstrap.cc:492] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0: No bootstrap required, opened a new log
I20260928 04:48:53.090622 8414 ts_tablet_manager.cc:1397] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 04:48:53.090736 8414 raft_consensus.cc:348] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.090802 8414 raft_consensus.cc:374] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:53.090822 8414 raft_consensus.cc:729] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c292c19bceb425ba2a232bbc4978ca0, State: Initialized, Role: FOLLOWER
I20260928 04:48:53.090867 8414 consensus_queue.cc:260] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.090956 8414 ts_tablet_manager.cc:1428] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:53.091148 8411 tablet_bootstrap.cc:492] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc: Bootstrap starting.
I20260928 04:48:53.091226 7887 catalog_manager.cc:4841] TS 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317): tablet 956ce34b2c3d4372a84b494d0df74a17 (table test-table [id=fbad8274724b40e88dc78ffe0edf88e0]) successfully deleted
I20260928 04:48:53.091406 7886 catalog_manager.cc:4841] TS 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2:42153): tablet 956ce34b2c3d4372a84b494d0df74a17 (table test-table [id=fbad8274724b40e88dc78ffe0edf88e0]) successfully deleted
I20260928 04:48:53.091480 8177 tablet_service.cc:1467] Processing CreateTablet for tablet b421cab88412405f8037dfc0dcc2c8a3 (DEFAULT_TABLE table=test-table [id=e57e3ec74f7f4b52a8decec666bbbe81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:48:53.091524 8411 tablet_bootstrap.cc:654] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:53.091567 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b421cab88412405f8037dfc0dcc2c8a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:53.092139 8479 ts_tablet_manager.cc:1905] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:53.092737 8412 tablet_bootstrap.cc:492] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7: Bootstrap starting.
I20260928 04:48:53.093024 8412 tablet_bootstrap.cc:654] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:53.093566 8411 tablet_bootstrap.cc:492] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc: No bootstrap required, opened a new log
I20260928 04:48:53.093585 8412 tablet_bootstrap.cc:492] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7: No bootstrap required, opened a new log
I20260928 04:48:53.093598 8479 ts_tablet_manager.cc:1918] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:48:53.093611 8411 ts_tablet_manager.cc:1397] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 04:48:53.093640 8479 log.cc:1196] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/956ce34b2c3d4372a84b494d0df74a17
I20260928 04:48:53.093639 8412 ts_tablet_manager.cc:1397] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:53.093755 8411 raft_consensus.cc:348] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.093748 8412 raft_consensus.cc:348] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.093814 8411 raft_consensus.cc:374] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:53.093818 8412 raft_consensus.cc:374] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:53.093834 8411 raft_consensus.cc:729] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf275ca2b6f453f9f116f06ff6431bc, State: Initialized, Role: FOLLOWER
I20260928 04:48:53.093842 8412 raft_consensus.cc:729] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600ab3a7dceb44ac877658947d3344d7, State: Initialized, Role: FOLLOWER
I20260928 04:48:53.093842 8479 ts_tablet_manager.cc:1939] T 956ce34b2c3d4372a84b494d0df74a17 P 9c292c19bceb425ba2a232bbc4978ca0: Deleting consensus metadata
I20260928 04:48:53.093874 8411 consensus_queue.cc:260] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.093888 8412 consensus_queue.cc:260] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.093950 8411 ts_tablet_manager.cc:1428] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:53.093988 8412 ts_tablet_manager.cc:1428] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:53.094460 7887 catalog_manager.cc:4841] TS 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537): tablet 956ce34b2c3d4372a84b494d0df74a17 (table test-table [id=fbad8274724b40e88dc78ffe0edf88e0]) successfully deleted
I20260928 04:48:53.176431 8454 raft_consensus.cc:482] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:48:53.176537 8454 raft_consensus.cc:504] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.176702 8454 leader_election.cc:288] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317), 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537)
I20260928 04:48:53.176913 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b421cab88412405f8037dfc0dcc2c8a3" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" is_pre_election: true
I20260928 04:48:53.176942 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b421cab88412405f8037dfc0dcc2c8a3" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c292c19bceb425ba2a232bbc4978ca0" is_pre_election: true
I20260928 04:48:53.177021 8337 raft_consensus.cc:2454] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 0.
I20260928 04:48:53.177022 8056 raft_consensus.cc:2454] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 0.
I20260928 04:48:53.177202 8131 leader_election.cc:302] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 600ab3a7dceb44ac877658947d3344d7, 9c292c19bceb425ba2a232bbc4978ca0; no voters:
I20260928 04:48:53.177312 8454 raft_consensus.cc:2781] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:48:53.177381 8454 raft_consensus.cc:482] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:48:53.177402 8454 raft_consensus.cc:3037] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:53.178273 8454 raft_consensus.cc:504] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.178390 8454 leader_election.cc:288] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 election: Requested vote from peers 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317), 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537)
I20260928 04:48:53.178529 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b421cab88412405f8037dfc0dcc2c8a3" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf275ca2b6f453f9f116f06ff6431bc"
I20260928 04:48:53.178548 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b421cab88412405f8037dfc0dcc2c8a3" candidate_uuid: "600ab3a7dceb44ac877658947d3344d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c292c19bceb425ba2a232bbc4978ca0"
I20260928 04:48:53.178618 8337 raft_consensus.cc:3037] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:53.178620 8056 raft_consensus.cc:3037] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:53.179219 8056 raft_consensus.cc:2454] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 1.
I20260928 04:48:53.179368 8337 raft_consensus.cc:2454] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 600ab3a7dceb44ac877658947d3344d7 in term 1.
I20260928 04:48:53.179392 8131 leader_election.cc:302] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf275ca2b6f453f9f116f06ff6431bc, 600ab3a7dceb44ac877658947d3344d7; no voters:
I20260928 04:48:53.179499 8454 raft_consensus.cc:2781] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:53.179548 8454 raft_consensus.cc:686] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 1 LEADER]: Becoming Leader. State: Replica: 600ab3a7dceb44ac877658947d3344d7, State: Running, Role: LEADER
I20260928 04:48:53.179596 8454 consensus_queue.cc:237] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } }
I20260928 04:48:53.180208 7900 catalog_manager.cc:5495] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "600ab3a7dceb44ac877658947d3344d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "600ab3a7dceb44ac877658947d3344d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 } health_report { overall_health: UNKNOWN } } }
I20260928 04:48:53.192206 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:53.190433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33055 (local address 127.0.98.2:42153)
0928 04:48:53.190481 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:53.190483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:53.190490 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:48:53.190507 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:53.190508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:53.190515 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:53.190561 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:48:53.190646 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:53.191100 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:53.191719 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:53.191882 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:53.191929 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:53.192027 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:48:53.192049 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:48:53.192084 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:53.192126 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 04:48:53.192138 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:48:53.192250 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:53.190270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42153 (local address 127.0.98.2:33055)
0928 04:48:53.190322 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:53.190334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:53.190376 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:53.190571 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:53.190571 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:53.190572 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 04:48:53.190607 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:53.190608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:53.191147 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:53.191148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:53.191642 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:53.191644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:53.191937 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:53.191937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:53.191971 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:53.191976 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 04:48:53.192092 (+ 116us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 04:48:53.192093 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 04:48:53.192125 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 04:48:53.192162 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 04:48:53.194204 8337 raft_consensus.cc:1261] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Refusing update from remote peer 600ab3a7dceb44ac877658947d3344d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:53.194204 8056 raft_consensus.cc:1261] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Refusing update from remote peer 600ab3a7dceb44ac877658947d3344d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:48:53.194423 8417 consensus_queue.cc:1035] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf275ca2b6f453f9f116f06ff6431bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:53.194551 8465 consensus_queue.cc:1035] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c292c19bceb425ba2a232bbc4978ca0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:48:53.204041 7900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:53.204130 7900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:48:53.204694 7900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89a2786edeb64703a90df4e9c4d26626: Sending DeleteTablet for 3 replicas of tablet b421cab88412405f8037dfc0dcc2c8a3
I20260928 04:48:53.204941 8037 tablet_service.cc:1514] Processing DeleteTablet for tablet b421cab88412405f8037dfc0dcc2c8a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54676
I20260928 04:48:53.204972 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet b421cab88412405f8037dfc0dcc2c8a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39890
I20260928 04:48:53.204972 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet b421cab88412405f8037dfc0dcc2c8a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:48:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36956
I20260928 04:48:53.205049 8480 tablet_replica.cc:330] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc: stopping tablet replica
I20260928 04:48:53.205049 8482 tablet_replica.cc:330] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7: stopping tablet replica
I20260928 04:48:53.205061 8479 tablet_replica.cc:330] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0: stopping tablet replica
I20260928 04:48:53.205096 8482 raft_consensus.cc:2229] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:48:53.205101 8480 raft_consensus.cc:2229] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:53.205116 8479 raft_consensus.cc:2229] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:48:53.205132 8480 raft_consensus.cc:2258] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:53.205142 8479 raft_consensus.cc:2258] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:53.205149 8482 raft_consensus.cc:2258] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:48:53.205437 8480 ts_tablet_manager.cc:1905] T b421cab88412405f8037dfc0dcc2c8a3 P 1bf275ca2b6f453f9f116f06ff6431bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:53.205477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 7972
I20260928 04:48:53.205523 8479 ts_tablet_manager.cc:1905] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:53.206982 8479 ts_tablet_manager.cc:1918] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:53.207021 8479 log.cc:1196] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/b421cab88412405f8037dfc0dcc2c8a3
I20260928 04:48:53.207172 8479 ts_tablet_manager.cc:1939] T b421cab88412405f8037dfc0dcc2c8a3 P 9c292c19bceb425ba2a232bbc4978ca0: Deleting consensus metadata
I20260928 04:48:53.207461 7887 catalog_manager.cc:4841] TS 9c292c19bceb425ba2a232bbc4978ca0 (127.0.98.3:43537): tablet b421cab88412405f8037dfc0dcc2c8a3 (table test-table [id=e57e3ec74f7f4b52a8decec666bbbe81]) successfully deleted
I20260928 04:48:53.209472 8482 ts_tablet_manager.cc:1905] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:48:53.211236 8482 ts_tablet_manager.cc:1918] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 04:48:53.211306 8482 log.cc:1196] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/b421cab88412405f8037dfc0dcc2c8a3
I20260928 04:48:53.211519 8482 ts_tablet_manager.cc:1939] T b421cab88412405f8037dfc0dcc2c8a3 P 600ab3a7dceb44ac877658947d3344d7: Deleting consensus metadata
I20260928 04:48:53.211896 7886 catalog_manager.cc:4841] TS 600ab3a7dceb44ac877658947d3344d7 (127.0.98.2:42153): tablet b421cab88412405f8037dfc0dcc2c8a3 (table test-table [id=e57e3ec74f7f4b52a8decec666bbbe81]) successfully deleted
W20260928 04:48:53.212528 7887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39317 (error 108)
I20260928 04:48:53.212833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 8112
I20260928 04:48:53.213050 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:53.212877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.0.1:54684)
0928 04:48:53.212968 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:53.213006 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":39,"threads_started":1}
W20260928 04:48:53.213128 7887 catalog_manager.cc:4584] TS 1bf275ca2b6f453f9f116f06ff6431bc (127.0.98.1:39317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b421cab88412405f8037dfc0dcc2c8a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39317: connect: Connection refused (error 111)
I20260928 04:48:53.219530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 8252
I20260928 04:48:53.224623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 7868
2026-09-28T04:48:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2785 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:53 dist-test-slave-jccs krb5kdc[8512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:53 dist-test-slave-jccs krb5kdc[8512](info): set up 2 sockets
Sep 28 04:48:53 dist-test-slave-jccs krb5kdc[8512](info): commencing operation
krb5kdc: starting...
W20260928 04:48:54.237890 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:54 dist-test-slave-jccs krb5kdc[8512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:54.350066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:54.426990 8525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:54.427217 8525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:54.427251 8525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:54.428301 8525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:54.428382 8525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:54.428406 8525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:54.428424 8525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:54.428452 8525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:54.429788 8525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:54.430034 8525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:54.430228 8525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:54.434643 8533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:54.434667 8531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:54.434643 8530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:54.434857 8525 server_base.cc:1034] running on GCE node
I20260928 04:48:54.435117 8525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:54.435331 8525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:54.436466 8525 hybrid_clock.cc:648] HybridClock initialized: now 1790570934436449 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T04:48:54Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1211 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:48:54 dist-test-slave-jccs krb5kdc[8538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:48:54 dist-test-slave-jccs krb5kdc[8538](info): set up 2 sockets
Sep 28 04:48:54 dist-test-slave-jccs krb5kdc[8538](info): commencing operation
krb5kdc: starting...
W20260928 04:48:55.429602 392 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:48:55 dist-test-slave-jccs krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:48:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:48:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:48:55.539602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41355
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:55.619119 8551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:55.619303 8551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:55.619341 8551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:55.620352 8551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:55.620383 8551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:55.620396 8551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:55.620409 8551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:55.620419 8551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:55.621779 8551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:55.621942 8551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:55.622090 8551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:55.626488 8556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:55.626596 8557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:55.626717 8559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:55.626812 8551 server_base.cc:1034] running on GCE node
I20260928 04:48:55.627048 8551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:55.627349 8551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:55.628504 8551 hybrid_clock.cc:648] HybridClock initialized: now 1790570935628493 us; error 43 us; skew 500 ppm
Sep 28 04:48:55 dist-test-slave-jccs krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:55.634053 8551 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:55.634523 8551 webserver.cc:460] Webserver started at http://127.0.98.62:33503/ using document root <none> and password file <none>
I20260928 04:48:55.634761 8551 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:55.634815 8551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:55.634938 8551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:55.636291 8551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "730ec0d66a8a84af2e44ceb7ca09eed7"
server_key_iv: "444bc2cc7a4d108df9a2bd60bdb4066a"
server_key_version: "encryptionkey@0"
I20260928 04:48:55.636719 8551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "730ec0d66a8a84af2e44ceb7ca09eed7"
server_key_iv: "444bc2cc7a4d108df9a2bd60bdb4066a"
server_key_version: "encryptionkey@0"
I20260928 04:48:55.638305 8551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 04:48:55.638990 8566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:55.639181 8551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:55.639272 8551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "730ec0d66a8a84af2e44ceb7ca09eed7"
server_key_iv: "444bc2cc7a4d108df9a2bd60bdb4066a"
server_key_version: "encryptionkey@0"
I20260928 04:48:55.639353 8551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:55.669682 8551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:55.671996 8551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:55.672161 8551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:55.676263 8551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41355
I20260928 04:48:55.676299 8618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41355 every 8 connection(s)
I20260928 04:48:55.676818 8551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:55.677531 8619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:48:55.679666 8619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55: Bootstrap starting.
I20260928 04:48:55.680182 8619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55: Neither blocks nor log segments found. Creating new log.
I20260928 04:48:55.680418 8619 log.cc:824] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:55.681015 8619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55: No bootstrap required, opened a new log
I20260928 04:48:55.682560 8619 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260928 04:48:55.682703 8619 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:48:55.682723 8619 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d72b2daffe4bd0b8a88beb4a14ca55, State: Initialized, Role: FOLLOWER
I20260928 04:48:55.682868 8619 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260928 04:48:55.682964 8619 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:55.682986 8619 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:55.683032 8619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:48:55.683837 8619 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260928 04:48:55.683916 8619 leader_election.cc:302] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2d72b2daffe4bd0b8a88beb4a14ca55; no voters:
I20260928 04:48:55.684003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 8551
I20260928 04:48:55.684088 8619 leader_election.cc:288] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:48:55.684132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:48:55.684140 8622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:48:55.684252 8622 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 1 LEADER]: Becoming Leader. State: Replica: e2d72b2daffe4bd0b8a88beb4a14ca55, State: Running, Role: LEADER
I20260928 04:48:55.684321 392 external_mini_cluster.cc:1428] Setting key 5924eafc40a0ae85046ee49de023c4fd
I20260928 04:48:55.684324 8619 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:55.684326 8622 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260928 04:48:55.684664 8622 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d72b2daffe4bd0b8a88beb4a14ca55. Latest consensus state: current_term: 1 leader_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } } }
I20260928 04:48:55.684729 8622 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:55.684877 8623 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } } }
I20260928 04:48:55.684980 8623 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:55.685148 8627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:55.685854 8627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:55.687475 8627 catalog_manager.cc:1347] Generated new cluster ID: 33720aad29354d6491fbece93daa180c
I20260928 04:48:55.687526 8627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:48:55 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:55.692976 8627 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:48:55.693605 8627 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:55.695254 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:55.684956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.0.98.62:41355)
0928 04:48:55.685109 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:55.685119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:55.685156 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:48:55.685311 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:55.685321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:55.685385 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:55.685543 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:55.686532 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:55.687889 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:55.688664 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:55.689074 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:55.689214 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:55.691449 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:55.691453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:55.691453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:55.691470 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:55.693785 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:55.694135 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:55.694137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:55.694138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:55.694225 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:55.694424 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:55.694425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:55.694425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:55.694629 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:55.694802 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:55.694906 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 04:48:55.695064 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260928 04:48:55.697005 8627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55: Generated new TSK 0
I20260928 04:48:55.697134 8627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:48:55.728229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41355
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:55.818861 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:55.819041 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:55.819063 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:55.820209 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:55.820248 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:55.820273 8645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:55.821781 8645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:55.822001 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:55.822232 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:55.827723 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:55.827739 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:55.827991 8645 server_base.cc:1034] running on GCE node
W20260928 04:48:55.827740 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:55.828271 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:55.828496 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:55.829670 8645 hybrid_clock.cc:648] HybridClock initialized: now 1790570935829655 us; error 29 us; skew 500 ppm
Sep 28 04:48:55 dist-test-slave-jccs krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:55.832075 8645 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:48:55.832465 8645 webserver.cc:460] Webserver started at http://127.0.98.1:41269/ using document root <none> and password file <none>
I20260928 04:48:55.832662 8645 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:55.832721 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:55.832852 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:55.833909 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b602fd3b923416b99c6cabf64336c7a"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "4dd10d1e981ed9b7ea1bc58e1bf0dfda"
server_key_iv: "01264a0d9d80276048b010deab0a035f"
server_key_version: "encryptionkey@0"
I20260928 04:48:55.834231 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b602fd3b923416b99c6cabf64336c7a"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "4dd10d1e981ed9b7ea1bc58e1bf0dfda"
server_key_iv: "01264a0d9d80276048b010deab0a035f"
server_key_version: "encryptionkey@0"
I20260928 04:48:55.835551 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:55.836230 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:55.836388 8645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:55.836550 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "4b602fd3b923416b99c6cabf64336c7a"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "4dd10d1e981ed9b7ea1bc58e1bf0dfda"
server_key_iv: "01264a0d9d80276048b010deab0a035f"
server_key_version: "encryptionkey@0"
I20260928 04:48:55.836614 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:55.871659 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:55.874166 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:55.874328 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:55.874580 8645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:55.874887 8645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:55.874935 8645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:55.874969 8645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:55.874989 8645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:55.880262 8645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41761
I20260928 04:48:55.880332 8773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41761 every 8 connection(s)
I20260928 04:48:55.880649 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:48:55.882926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 8645
I20260928 04:48:55.883013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:48:55.883203 392 external_mini_cluster.cc:1428] Setting key 67fb2734b234f39dc031efa431daf5f0
Sep 28 04:48:55 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:55.890326 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:55.881433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55617 (local address 127.0.98.62:41355)
0928 04:48:55.881490 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:55.881493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:55.881702 (+ 209us) server_negotiation.cc:407] Connection header received
0928 04:48:55.882514 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:55.882515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:55.882527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:55.882576 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:55.883192 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:55.883951 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:55.885041 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:55.885309 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:55.885414 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:55.887550 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:55.887551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:55.887552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:55.887559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:55.888915 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:55.889380 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:55.889381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:55.889382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:55.889439 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:55.889703 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:55.889704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:55.889704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:55.889824 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:55.889899 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:55.890154 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 04:48:55.890236 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:48:55.890517 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:55.881414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:55617)
0928 04:48:55.881588 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:55.881620 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:55.882422 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:55.882626 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:55.882629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:55.882860 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:55.883096 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:55.883105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:55.884044 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:55.884045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:55.884957 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:55.884959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:55.885484 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:55.885485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:55.885602 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:55.886090 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:55.886098 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:55.887457 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:55.889022 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:55.889029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:55.889031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:55.889310 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:55.889512 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:55.889512 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:55.889513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:55.889622 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:55.889890 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:55.889896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:55.890030 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 04:48:55.890133 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 04:48:55.890278 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260928 04:48:55.890987 8774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41355
I20260928 04:48:55.891062 8774 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:55.891185 8774 heartbeater.cc:511] Master 127.0.98.62:41355 requested a full tablet report, sending...
I20260928 04:48:55.891678 8583 ts_manager.cc:194] Registered new tserver with Master: 4b602fd3b923416b99c6cabf64336c7a (127.0.98.1:41761)
I20260928 04:48:55.892266 8583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:48:55.907558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41355
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:55.983573 8779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:55.983760 8779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:55.983781 8779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:55.984895 8779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:55.984954 8779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:55.984999 8779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:55.986496 8779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:55.986734 8779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:55.986943 8779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:55.991595 8784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:55.991585 8787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:55.991586 8785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:55.991897 8779 server_base.cc:1034] running on GCE node
I20260928 04:48:55.992129 8779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:55.992350 8779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:55.993510 8779 hybrid_clock.cc:648] HybridClock initialized: now 1790570935993489 us; error 36 us; skew 500 ppm
Sep 28 04:48:55 dist-test-slave-jccs krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:56.000329 8779 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:48:56.000787 8779 webserver.cc:460] Webserver started at http://127.0.98.2:45319/ using document root <none> and password file <none>
I20260928 04:48:56.001016 8779 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:56.001082 8779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:56.001263 8779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:56.002281 8779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "8930b1d908d3489da94857a89d3d6dff"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "57bbb1cf5c4f9e424ab2b3a1f3406c9b"
server_key_iv: "ccac72866f7e83217213886f91b95764"
server_key_version: "encryptionkey@0"
I20260928 04:48:56.002628 8779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8930b1d908d3489da94857a89d3d6dff"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "57bbb1cf5c4f9e424ab2b3a1f3406c9b"
server_key_iv: "ccac72866f7e83217213886f91b95764"
server_key_version: "encryptionkey@0"
I20260928 04:48:56.003873 8779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:56.004597 8794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.004787 8779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:56.004870 8779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "8930b1d908d3489da94857a89d3d6dff"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "57bbb1cf5c4f9e424ab2b3a1f3406c9b"
server_key_iv: "ccac72866f7e83217213886f91b95764"
server_key_version: "encryptionkey@0"
I20260928 04:48:56.004922 8779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:56.028798 8779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:56.031170 8779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:56.031282 8779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:56.031454 8779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:56.031814 8779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:56.031858 8779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.031896 8779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:56.031922 8779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.037606 8779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40309
I20260928 04:48:56.037712 8907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40309 every 8 connection(s)
I20260928 04:48:56.037985 8779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:48:56.041976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 8779
I20260928 04:48:56.042098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:48:56.042344 392 external_mini_cluster.cc:1428] Setting key 7d919be57665b4686098998bd96a46b1
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:56.047451 8910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.038641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:59991)
0928 04:48:56.038800 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.038834 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.039774 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.040013 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.040018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.040274 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.040459 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.040463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.041110 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.041111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.042002 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.042005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.042454 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.042455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.042574 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.043094 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.043102 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.044394 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:56.046213 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:56.046218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:56.046220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:56.046426 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:56.046632 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:56.046633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:56.046633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:56.046724 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:56.046997 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:56.047001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:56.047112 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 04:48:56.047190 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:48:56.047300 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260928 04:48:56.047480 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.038690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59991 (local address 127.0.98.62:41355)
0928 04:48:56.038765 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.038767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.038933 (+ 166us) server_negotiation.cc:407] Connection header received
0928 04:48:56.039864 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.039865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.039876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.039933 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.040530 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.041022 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.042114 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.042364 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.042446 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.044477 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:56.044479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:56.044479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:56.044487 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:56.046133 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:56.046499 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:56.046500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:56.046501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:56.046563 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:56.046790 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:56.046791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:56.046791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:56.046932 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:56.047005 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:56.047243 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 04:48:56.047340 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:48:56.047930 8908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41355
I20260928 04:48:56.048043 8908 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:56.048244 8908 heartbeater.cc:511] Master 127.0.98.62:41355 requested a full tablet report, sending...
I20260928 04:48:56.048632 8583 ts_manager.cc:194] Registered new tserver with Master: 8930b1d908d3489da94857a89d3d6dff (127.0.98.2:40309)
I20260928 04:48:56.048985 8583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59991
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:48:56.067373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41355
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:56.154945 8913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:56.155143 8913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:56.155181 8913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:56.156303 8913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:56.156383 8913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:56.156431 8913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:56.157925 8913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:56.158172 8913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:56.158375 8913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:56.162897 8921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:56.162942 8919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:56.162914 8918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:56.163160 8913 server_base.cc:1034] running on GCE node
I20260928 04:48:56.163429 8913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:56.163661 8913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:56.164820 8913 hybrid_clock.cc:648] HybridClock initialized: now 1790570936164806 us; error 31 us; skew 500 ppm
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:56.167284 8913 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:48:56.167685 8913 webserver.cc:460] Webserver started at http://127.0.98.3:33587/ using document root <none> and password file <none>
I20260928 04:48:56.167905 8913 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:56.167969 8913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:56.168113 8913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:48:56.169047 8913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "992c4f0306b34d47836054c262503e21"
format_stamp: "Formatted at 2026-09-28 04:48:56 on dist-test-slave-jccs"
server_key: "51dc87c7067c780f9921bcd07d25990f"
server_key_iv: "340634e2b1c6ce5cfdb75d949ff85645"
server_key_version: "encryptionkey@0"
I20260928 04:48:56.169422 8913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "992c4f0306b34d47836054c262503e21"
format_stamp: "Formatted at 2026-09-28 04:48:56 on dist-test-slave-jccs"
server_key: "51dc87c7067c780f9921bcd07d25990f"
server_key_iv: "340634e2b1c6ce5cfdb75d949ff85645"
server_key_version: "encryptionkey@0"
I20260928 04:48:56.170630 8913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:56.171334 8928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.171492 8913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:56.171583 8913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "992c4f0306b34d47836054c262503e21"
format_stamp: "Formatted at 2026-09-28 04:48:56 on dist-test-slave-jccs"
server_key: "51dc87c7067c780f9921bcd07d25990f"
server_key_iv: "340634e2b1c6ce5cfdb75d949ff85645"
server_key_version: "encryptionkey@0"
I20260928 04:48:56.171658 8913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:56.184721 8913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:56.187002 8913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:56.187168 8913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:56.187383 8913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:56.187685 8913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:56.187743 8913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.187785 8913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:56.187811 8913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.193372 8913 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44009
I20260928 04:48:56.193418 9041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44009 every 8 connection(s)
I20260928 04:48:56.193710 8913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:48:56.201709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 8913
I20260928 04:48:56.201818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:48:56.202013 392 external_mini_cluster.cc:1428] Setting key 7bf6aded2c565225b30b96fa570fb325
I20260928 04:48:56.202821 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.194423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35779 (local address 127.0.98.62:41355)
0928 04:48:56.194463 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.194465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.194681 (+ 216us) server_negotiation.cc:407] Connection header received
0928 04:48:56.195625 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.195626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.195639 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.195689 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.196276 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.196800 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.197896 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.198109 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.198187 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.200158 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:56.200160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:56.200160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:56.200167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:56.201354 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:56.201815 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:56.201816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:56.201816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:56.201899 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:56.202182 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:56.202183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:56.202183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:56.202308 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:56.202385 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:56.202653 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 04:48:56.202732 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:48:56.203008 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.194408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:35779)
0928 04:48:56.194571 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.194605 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.195504 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.195755 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.195760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.196005 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.196186 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.196190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.196893 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.196894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.197807 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.197810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.198200 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.198201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.198315 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.198795 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.198802 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.200070 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:48:56.201423 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:56.201428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:56.201429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:56.201722 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:56.201969 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:48:56.201970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:48:56.201970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:48:56.202111 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:48:56.202378 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:48:56.202384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:48:56.202524 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 04:48:56.202631 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 04:48:56.202778 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260928 04:48:56.203377 9042 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41355
I20260928 04:48:56.203456 9042 heartbeater.cc:464] Registering TS with master...
I20260928 04:48:56.203593 9042 heartbeater.cc:511] Master 127.0.98.62:41355 requested a full tablet report, sending...
I20260928 04:48:56.203920 8583 ts_manager.cc:194] Registered new tserver with Master: 992c4f0306b34d47836054c262503e21 (127.0.98.3:44009)
I20260928 04:48:56.204303 8583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35779
I20260928 04:48:56.213588 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 04:48:56.277493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 8645
I20260928 04:48:56.281543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 8779
I20260928 04:48:56.285898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 8913
I20260928 04:48:56.289882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 8551
I20260928 04:48:56.294893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41355
--webserver_interface=127.0.98.62
--webserver_port=33503
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:48:56.370950 9051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:56.371156 9051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:56.371177 9051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:56.372295 9051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:48:56.372334 9051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:56.372347 9051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:56.372359 9051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:48:56.372411 9051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:48:56.373850 9051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:56.374064 9051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:56.374277 9051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:56.379014 9056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:56.379052 9057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:56.379025 9059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:56.379169 9051 server_base.cc:1034] running on GCE node
I20260928 04:48:56.379374 9051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:56.379652 9051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:56.380810 9051 hybrid_clock.cc:648] HybridClock initialized: now 1790570936380784 us; error 38 us; skew 500 ppm
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:56.383086 9051 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:48:56.383515 9051 webserver.cc:460] Webserver started at http://127.0.98.62:33503/ using document root <none> and password file <none>
I20260928 04:48:56.383733 9051 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:56.383808 9051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:56.385005 9051 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:48:56.385697 9066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.385859 9051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.385953 9051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "730ec0d66a8a84af2e44ceb7ca09eed7"
server_key_iv: "444bc2cc7a4d108df9a2bd60bdb4066a"
server_key_version: "encryptionkey@0"
I20260928 04:48:56.386265 9051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:56.408354 9051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:56.410125 9051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:56.410286 9051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:56.414103 9051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41355
I20260928 04:48:56.414147 9118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41355 every 8 connection(s)
I20260928 04:48:56.414609 9051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:48:56.415674 9119 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:48:56.416217 9119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55: Bootstrap starting.
I20260928 04:48:56.419137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 9051
I20260928 04:48:56.419380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41761
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41355
--webserver_port=41269
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:48:56.423700 9119 log.cc:824] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55: Log is configured to *not* fsync() on all Append() calls
I20260928 04:48:56.425136 9119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:48:56.425436 9119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55: Bootstrap complete.
I20260928 04:48:56.427533 9119 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260928 04:48:56.427776 9119 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d72b2daffe4bd0b8a88beb4a14ca55, State: Initialized, Role: FOLLOWER
I20260928 04:48:56.427924 9119 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260928 04:48:56.427991 9119 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:48:56.428049 9119 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:48:56.428088 9119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:48:56.428992 9119 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260928 04:48:56.429097 9119 leader_election.cc:302] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2d72b2daffe4bd0b8a88beb4a14ca55; no voters:
I20260928 04:48:56.429282 9119 leader_election.cc:288] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:48:56.429340 9123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:48:56.429462 9123 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [term 2 LEADER]: Becoming Leader. State: Replica: e2d72b2daffe4bd0b8a88beb4a14ca55, State: Running, Role: LEADER
I20260928 04:48:56.429530 9119 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:48:56.429574 9123 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } }
I20260928 04:48:56.429801 9124 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } } }
I20260928 04:48:56.429878 9124 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:56.429798 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d72b2daffe4bd0b8a88beb4a14ca55. Latest consensus state: current_term: 2 leader_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d72b2daffe4bd0b8a88beb4a14ca55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41355 } } }
I20260928 04:48:56.430025 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55 [sys.catalog]: This master's current role is: LEADER
I20260928 04:48:56.430126 9128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:48:56.430768 9128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:48:56.430886 9128 catalog_manager.cc:1259] Loaded cluster ID: 33720aad29354d6491fbece93daa180c
I20260928 04:48:56.430932 9128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:48:56.432178 9128 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:48:56.432333 9128 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e2d72b2daffe4bd0b8a88beb4a14ca55: Loaded TSK: 0
I20260928 04:48:56.432547 9128 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:48:56.498682 9121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:56.498901 9121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:56.498925 9121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:56.500041 9121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:56.500077 9121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:56.500101 9121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:48:56.501473 9121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41269
--tserver_master_addrs=127.0.98.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:56.501705 9121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:56.501924 9121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:56.507339 9145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:56.507345 9146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:56.507402 9148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:56.507601 9121 server_base.cc:1034] running on GCE node
I20260928 04:48:56.507790 9121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:56.508030 9121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:56.509222 9121 hybrid_clock.cc:648] HybridClock initialized: now 1790570936509198 us; error 35 us; skew 500 ppm
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570936, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:56.511847 9121 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 04:48:56.512323 9121 webserver.cc:460] Webserver started at http://127.0.98.1:41269/ using document root <none> and password file <none>
I20260928 04:48:56.512528 9121 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:56.512600 9121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:56.513943 9121 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:48:56.514549 9155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.514688 9121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:56.514761 9121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "4b602fd3b923416b99c6cabf64336c7a"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "4dd10d1e981ed9b7ea1bc58e1bf0dfda"
server_key_iv: "01264a0d9d80276048b010deab0a035f"
server_key_version: "encryptionkey@0"
I20260928 04:48:56.515026 9121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:56.525604 9121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:56.527807 9121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:56.527958 9121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:56.528182 9121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:56.528510 9121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:56.528560 9121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.528599 9121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:56.528625 9121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.535642 9121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41761
I20260928 04:48:56.535709 9268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41761 every 8 connection(s)
I20260928 04:48:56.536020 9121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:56.543812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 9121
I20260928 04:48:56.544106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40309
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41355
--webserver_port=45319
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:48:56.544658 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.536718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:60581)
0928 04:48:56.536877 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.536913 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.537423 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.537840 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.537843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.538345 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.538514 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.538518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.539998 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.539998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.540758 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.540759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.541541 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.541542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.541629 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.542100 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.542107 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.544489 (+ 2382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
W20260928 04:48:56.545094 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:56.536773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60581 (local address 127.0.98.62:41355)
0928 04:48:56.537277 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.537285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.537318 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:48:56.537606 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.537613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.537662 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.537806 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.538625 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.539909 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.540855 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.541466 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.541564 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.544623 (+ 3059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60581: BlockingRecv error: recv got EOF from 127.0.98.1:60581 (error 108)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":97,"threads_started":1}
W20260928 04:48:56.545657 9269 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41355 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41355: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:56.551196 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.545952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:60859)
0928 04:48:56.545995 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.546005 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.546038 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.546174 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.546175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.546329 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.546377 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.546379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.547259 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.547260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.548140 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.548143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.548584 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.548585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.548658 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.549079 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.549084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.551106 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 04:48:56.551555 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:56.546006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60859 (local address 127.0.98.62:41355)
0928 04:48:56.546041 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.546043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.546076 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:48:56.546094 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.546095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.546105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.546156 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.546462 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.547175 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.548231 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.548513 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.548570 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.551477 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60859: BlockingRecv error: recv got EOF from 127.0.98.1:60859 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:56.556455 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.551461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:33405)
0928 04:48:56.551632 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.551642 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.551677 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.551912 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.551913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.552054 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.552082 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.552083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.553037 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.553037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.553662 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.553664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.554138 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.554139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.554186 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.554599 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.554604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.556394 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157}
W20260928 04:48:56.556543 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:56.551535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33405 (local address 127.0.98.62:41355)
0928 04:48:56.551779 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.551781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.551791 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:48:56.551813 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.551813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.551824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.551892 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.552177 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.552939 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.553745 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.554058 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.554141 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.556467 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33405: BlockingRecv error: recv got EOF from 127.0.98.1:33405 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":69,"threads_started":1}
W20260928 04:48:56.556562 9269 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:48:56.625877 9273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:56.626066 9273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:56.626089 9273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:56.627183 9273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:56.627218 9273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:56.627243 9273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:48:56.628533 9273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45319
--tserver_master_addrs=127.0.98.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:56.628708 9273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:56.628899 9273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:56.633607 9282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:56.633600 9280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:56.633680 9279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:56.633826 9273 server_base.cc:1034] running on GCE node
I20260928 04:48:56.634004 9273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:56.634212 9273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:56.635341 9273 hybrid_clock.cc:648] HybridClock initialized: now 1790570936635324 us; error 31 us; skew 500 ppm
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570936, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:56.637594 9273 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 04:48:56.638132 9273 webserver.cc:460] Webserver started at http://127.0.98.2:45319/ using document root <none> and password file <none>
I20260928 04:48:56.638384 9273 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:56.638464 9273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:56.640295 9273 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:48:56.641014 9289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.641258 9273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:56.641336 9273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "8930b1d908d3489da94857a89d3d6dff"
format_stamp: "Formatted at 2026-09-28 04:48:55 on dist-test-slave-jccs"
server_key: "57bbb1cf5c4f9e424ab2b3a1f3406c9b"
server_key_iv: "ccac72866f7e83217213886f91b95764"
server_key_version: "encryptionkey@0"
I20260928 04:48:56.641618 9273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:56.681332 9273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:56.683068 9273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:56.683171 9273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:56.683349 9273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:56.683722 9273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:56.683779 9273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.683813 9273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:56.683856 9273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.689749 9273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40309
I20260928 04:48:56.689790 9402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40309 every 8 connection(s)
I20260928 04:48:56.690138 9273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 04:48:56.697552 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:56.690842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45117 (local address 127.0.98.62:41355)
0928 04:48:56.690905 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.690906 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.691150 (+ 244us) server_negotiation.cc:407] Connection header received
0928 04:48:56.691729 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.691730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.691746 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.691800 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.692754 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.693318 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.694328 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.694529 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.694636 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.697497 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45117: BlockingRecv error: recv got EOF from 127.0.98.2:45117 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 04:48:56.697620 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.690832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:45117)
0928 04:48:56.691012 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.691073 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.691572 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.691866 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.691870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.692458 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.692641 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.692645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.693421 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.693422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.694224 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.694226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.694646 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.694647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.694773 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.695263 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.695271 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.697414 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
W20260928 04:48:56.697824 9403 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41355 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41355: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260928 04:48:56.699797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 9273
I20260928 04:48:56.700090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44009
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41355
--webserver_port=33587
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:56.702064 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.698076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:34879)
0928 04:48:56.698124 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.698134 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.698170 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.698287 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.698288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.698407 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.698444 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.698445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.699033 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.699034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.699559 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.699561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.699915 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.699915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.699974 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.700404 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.700408 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.701992 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:48:56.702114 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:56.698135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34879 (local address 127.0.98.62:41355)
0928 04:48:56.698177 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.698179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.698188 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:56.698224 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.698225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.698234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.698283 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.698515 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.698946 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.699639 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.699834 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.699914 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.702053 (+ 2139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34879: BlockingRecv error: recv got EOF from 127.0.98.2:34879 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:56.706801 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.702366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:51331)
0928 04:48:56.702475 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.702486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.702538 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.702661 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.702662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.702806 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.702858 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.702859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.703418 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.703419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.704293 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.704295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.704596 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.704597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.704665 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.704994 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.705000 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.706732 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260928 04:48:56.706854 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:56.702432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51331 (local address 127.0.98.62:41355)
0928 04:48:56.702485 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.702487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.702524 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:48:56.702588 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.702589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.702596 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.702637 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.702928 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.703364 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.704374 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.704530 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.704574 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.706799 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51331: BlockingRecv error: recv got EOF from 127.0.98.2:51331 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 04:48:56.706899 9403 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:48:56.777961 9406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:48:56.778187 9406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:48:56.778224 9406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:48:56.779309 9406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:48:56.779371 9406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:48:56.779408 9406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:48:56.780721 9406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33587
--tserver_master_addrs=127.0.98.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:48:56.780982 9406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:48:56.781220 9406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:48:56.785853 9414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:56.785907 9411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:48:56.785894 9412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:48:56.786033 9406 server_base.cc:1034] running on GCE node
I20260928 04:48:56.786235 9406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:48:56.786417 9406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:48:56.787595 9406 hybrid_clock.cc:648] HybridClock initialized: now 1790570936787581 us; error 47 us; skew 500 ppm
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570936, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:48:56.790030 9406 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 04:48:56.790473 9406 webserver.cc:460] Webserver started at http://127.0.98.3:33587/ using document root <none> and password file <none>
I20260928 04:48:56.790694 9406 fs_manager.cc:359] Metadata directory not provided
I20260928 04:48:56.790771 9406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:48:56.791989 9406 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:48:56.792508 9421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.792673 9406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:48:56.792768 9406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "992c4f0306b34d47836054c262503e21"
format_stamp: "Formatted at 2026-09-28 04:48:56 on dist-test-slave-jccs"
server_key: "51dc87c7067c780f9921bcd07d25990f"
server_key_iv: "340634e2b1c6ce5cfdb75d949ff85645"
server_key_version: "encryptionkey@0"
I20260928 04:48:56.793090 9406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:48:56.806389 9406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:48:56.808100 9406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:48:56.808254 9406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:48:56.808466 9406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:48:56.808749 9406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:48:56.808800 9406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.808840 9406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:48:56.808897 9406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:48:56.814654 9406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44009
I20260928 04:48:56.814702 9534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44009 every 8 connection(s)
I20260928 04:48:56.815006 9406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 04:48:56.821460 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:56.815662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34101 (local address 127.0.98.62:41355)
0928 04:48:56.815728 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.815731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.815945 (+ 214us) server_negotiation.cc:407] Connection header received
0928 04:48:56.816380 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.816382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.816391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.816437 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.817327 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.817782 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.818697 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.818865 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.818913 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.821416 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34101: BlockingRecv error: recv got EOF from 127.0.98.3:34101 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:48:56.821470 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.815652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:34101)
0928 04:48:56.815834 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.815866 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.816305 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.816495 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.816498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.817045 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.817255 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.817259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.817867 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.817867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.818630 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.818632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.818927 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.818927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.819018 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.819387 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.819394 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.821342 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
W20260928 04:48:56.821750 9535 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41355 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41355: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:56.825073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 9406
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:56.826107 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.821997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:36149)
0928 04:48:56.822044 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.822054 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.822096 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.822307 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.822308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.822427 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.822463 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.822464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.823056 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.823057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.823537 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.823538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.823886 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.823886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.823950 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.824344 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.824348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.826036 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:48:56.826133 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:56.822169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36149 (local address 127.0.98.62:41355)
0928 04:48:56.822206 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.822207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.822212 (+ 5us) server_negotiation.cc:407] Connection header received
0928 04:48:56.822228 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.822228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.822235 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.822304 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.822540 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.822974 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.823613 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.823808 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.823886 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.826095 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36149: BlockingRecv error: recv got EOF from 127.0.98.3:36149 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:56 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:56.831285 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.826401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:56147)
0928 04:48:56.826484 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:56.826496 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:56.826549 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:56.826716 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:56.826717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:56.826821 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:56.826842 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.826843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.827590 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.827591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.828343 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:56.828344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.828789 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.828790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:56.828839 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.829442 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:56.829447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:56.831212 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 04:48:56.831331 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:56.826443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56147 (local address 127.0.98.62:41355)
0928 04:48:56.826484 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.826486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.826534 (+ 48us) server_negotiation.cc:407] Connection header received
0928 04:48:56.826619 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.826620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.826626 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.826712 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.826905 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.827530 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.828435 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.828717 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.828780 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.831270 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56147: BlockingRecv error: recv got EOF from 127.0.98.3:56147 (error 108)
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":24}
W20260928 04:48:56.831513 9535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:48:56.833451 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:56.825491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.0.98.62:41355)
0928 04:48:56.825551 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:56.825553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:56.825707 (+ 154us) server_negotiation.cc:407] Connection header received
0928 04:48:56.825798 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:56.825799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:56.825817 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:56.825880 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:56.826665 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.827391 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.828435 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:56.828717 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:56.828780 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:56.830261 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:48:56.830266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:48:56.830267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:48:56.830279 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:48:56.832080 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:56.832449 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:56.832452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:56.832453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:56.832563 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:48:56.832771 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:48:56.832772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:48:56.832772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:48:56.832988 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:48:56.833129 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 04:48:56.833266 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 04:48:56.833367 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 28 04:48:57 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:57 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:57.561115 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:57.556905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:35473)
0928 04:48:57.557147 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:57.557159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:57.557260 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:57.557402 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:57.557402 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:57.557554 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:57.557599 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:57.557601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.558201 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.558202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:57.558706 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:57.558708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.559006 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.559006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:57.559042 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:57.559458 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:57.559462 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:57.561050 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":66,"threads_started":1}
W20260928 04:48:57.561283 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:57.556944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35473 (local address 127.0.98.62:41355)
0928 04:48:57.557208 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:57.557210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:57.557241 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:48:57.557319 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:57.557320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:57.557328 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:57.557381 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:57.557657 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.558141 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.558775 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.558950 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.558996 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:57.561135 (+ 2139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35473: BlockingRecv error: recv got EOF from 127.0.98.1:35473 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":173,"threads_started":1}
Sep 28 04:48:57 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:57 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:57.711500 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:57.707212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:39705)
0928 04:48:57.707375 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:57.707395 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:57.707481 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:57.707648 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:57.707650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:57.707810 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:57.707852 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:57.707854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.708468 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.708469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:57.708963 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:57.708964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.709338 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.709339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:57.709396 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:57.709846 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:57.709850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:57.711437 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
W20260928 04:48:57.711556 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:57.707293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39705 (local address 127.0.98.62:41355)
0928 04:48:57.707364 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:57.707367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:57.707464 (+ 97us) server_negotiation.cc:407] Connection header received
0928 04:48:57.707556 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:57.707557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:57.707572 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:57.707642 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:57.707951 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.708397 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.709059 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.709271 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.709339 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:57.711515 (+ 2176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39705: BlockingRecv error: recv got EOF from 127.0.98.2:39705 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 04:48:57 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:57 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:57.836184 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:57.831890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:52747)
0928 04:48:57.831997 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:57.832007 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:57.832052 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:57.832165 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:57.832166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:57.832314 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:57.832357 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:57.832359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.832952 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.832953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:57.833528 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:57.833530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.833809 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.833809 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:57.833843 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:57.834227 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:57.834230 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:57.836120 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
W20260928 04:48:57.836290 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:57.831949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52747 (local address 127.0.98.62:41355)
0928 04:48:57.832029 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:57.832031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:57.832061 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:48:57.832096 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:57.832096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:57.832108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:57.832147 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:57.832417 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.832893 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.833593 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:57.833754 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:57.833800 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:57.836216 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52747: BlockingRecv error: recv got EOF from 127.0.98.3:52747 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 04:48:58 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:58 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:58.565632 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:58.561642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:59967)
0928 04:48:58.561787 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:58.561797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:58.561847 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:58.561987 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:58.561988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:58.562132 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:58.562176 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:58.562178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.562729 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.562730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:58.563192 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:58.563193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.563474 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.563474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:58.563516 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:58.563989 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:58.563993 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:58.565573 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
W20260928 04:48:58.565714 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:58.561735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59967 (local address 127.0.98.62:41355)
0928 04:48:58.561829 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:58.561831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:58.561839 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:48:58.561888 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:58.561888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:58.561911 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:58.561964 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:58.562235 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.562673 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.563254 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.563422 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.563466 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:58.565650 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59967: BlockingRecv error: recv got EOF from 127.0.98.1:59967 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":48,"threads_started":1}
Sep 28 04:48:58 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:58 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:58.716230 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:58.711927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:45707)
0928 04:48:58.712063 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:58.712091 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:58.712142 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:58.712255 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:58.712256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:58.712393 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:58.712457 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:58.712459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.713054 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.713055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:58.713580 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:58.713581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.713863 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.713864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:58.713896 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:58.714409 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:58.714415 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:58.716172 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
W20260928 04:48:58.716298 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:58.711987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45707 (local address 127.0.98.62:41355)
0928 04:48:58.712050 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:58.712052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:58.712142 (+ 90us) server_negotiation.cc:407] Connection header received
0928 04:48:58.712183 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:58.712184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:58.712193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:58.712243 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:58.712528 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.712983 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.713635 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.713799 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.713845 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:58.716251 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45707: BlockingRecv error: recv got EOF from 127.0.98.2:45707 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 04:48:58 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:58 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:58.841002 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:58.836631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:34299)
0928 04:48:58.836778 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:58.836788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:58.836871 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:58.837022 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:58.837023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:58.837219 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:58.837263 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:58.837265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.837871 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.837872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:58.838348 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:58.838350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.838682 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.838683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:58.838743 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:58.839203 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:58.839213 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:58.840943 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
W20260928 04:48:58.841162 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:58.836782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34299 (local address 127.0.98.62:41355)
0928 04:48:58.836853 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:58.836856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:58.836865 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:48:58.836928 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:58.836929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:58.836943 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:58.837016 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:58.837355 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.837792 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.838429 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:58.838605 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:58.838684 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:58.841029 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34299: BlockingRecv error: recv got EOF from 127.0.98.3:34299 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 04:48:59 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:59 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:59.570448 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:59.566214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:49925)
0928 04:48:59.566356 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:59.566366 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:59.566424 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:59.566549 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:59.566550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:59.566704 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:59.566752 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:59.566753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.567453 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.567454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:59.567951 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:59.567952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.568278 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.568278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:59.568314 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:59.568728 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:59.568732 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:59.570393 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
W20260928 04:48:59.570557 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:59.566270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49925 (local address 127.0.98.62:41355)
0928 04:48:59.566384 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:59.566386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:59.566418 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:48:59.566463 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:59.566464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:59.566473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:59.566537 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:59.566814 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.567389 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.568016 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.568224 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.568268 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:59.570477 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49925: BlockingRecv error: recv got EOF from 127.0.98.1:49925 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
Sep 28 04:48:59 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:59 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:59.721095 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:59.716695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:52425)
0928 04:48:59.716859 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:59.716868 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:59.716946 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:59.717101 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:59.717102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:59.717337 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:59.717379 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:59.717381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.717954 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.717955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:59.718436 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:59.718438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.718791 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.718792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:59.718846 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:59.719280 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:59.719284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:59.720994 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
W20260928 04:48:59.721148 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:59.716749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52425 (local address 127.0.98.62:41355)
0928 04:48:59.716813 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:59.716815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:59.716928 (+ 113us) server_negotiation.cc:407] Connection header received
0928 04:48:59.717010 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:59.717012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:59.717029 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:59.717095 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:59.717449 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.717879 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.718517 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.718710 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.718790 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:59.721081 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52425: BlockingRecv error: recv got EOF from 127.0.98.2:52425 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 04:48:59 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:48:59 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:48:59.845808 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:48:59.841485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:47193)
0928 04:48:59.841616 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 04:48:59.841634 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 04:48:59.841688 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:48:59.841808 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:48:59.841809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:48:59.841952 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:48:59.841998 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:59.841999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.842735 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.842736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:59.843296 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:48:59.843298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.843641 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.843642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:48:59.843719 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:59.844128 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:48:59.844132 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:48:59.845753 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
W20260928 04:48:59.845929 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:48:59.841558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47193 (local address 127.0.98.62:41355)
0928 04:48:59.841624 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:48:59.841626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:48:59.841688 (+ 62us) server_negotiation.cc:407] Connection header received
0928 04:48:59.841728 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:48:59.841729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:48:59.841740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:48:59.841801 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:48:59.842048 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.842646 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.843374 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:48:59.843567 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:48:59.843641 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:48:59.845835 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47193: BlockingRecv error: recv got EOF from 127.0.98.3:47193 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 04:49:00 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:00 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:00.575495 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:00.571068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:37509)
0928 04:49:00.571235 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:00.571249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:00.571322 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:00.571466 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:00.571467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:00.571616 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:00.571679 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:00.571681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.572284 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.572285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:00.572783 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:00.572784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.573082 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.573082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:00.573118 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:00.573638 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:00.573642 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:00.575407 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
W20260928 04:49:00.575538 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:00.571176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37509 (local address 127.0.98.62:41355)
0928 04:49:00.571301 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:00.571303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:00.571311 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:49:00.571388 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:00.571389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:00.571398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:00.571443 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:00.571741 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.572227 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.572848 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.573024 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.573069 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:00.575473 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37509: BlockingRecv error: recv got EOF from 127.0.98.1:37509 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
Sep 28 04:49:00 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:00 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:00.729547 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:00.721523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:51549)
0928 04:49:00.721667 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:00.721683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:00.721769 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:00.721928 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:00.721929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:00.722094 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:00.722137 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:00.722139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.722767 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.722768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:00.723309 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:00.723311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.723647 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.723648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:00.723707 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:00.727751 (+ 4044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:00.727756 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:00.729455 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260928 04:49:00.729598 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:00.721587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51549 (local address 127.0.98.62:41355)
0928 04:49:00.721654 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:00.721657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:00.721749 (+ 92us) server_negotiation.cc:407] Connection header received
0928 04:49:00.721842 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:00.721843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:00.721855 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:00.721922 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:00.722213 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.722687 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.723389 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.723579 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.723649 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:00.729539 (+ 5890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51549: BlockingRecv error: recv got EOF from 127.0.98.2:51549 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 04:49:00 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:00 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:00.850762 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:00.846217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:39887)
0928 04:49:00.846401 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:00.846414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:00.846505 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:00.846677 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:00.846678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:00.846854 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:00.846899 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:00.846901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.847630 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.847631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:00.848145 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:00.848146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.848504 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.848505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:00.848562 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:00.849075 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:00.849079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:00.850708 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":105,"threads_started":1}
W20260928 04:49:00.850855 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:00.846293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39887 (local address 127.0.98.62:41355)
0928 04:49:00.846385 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:00.846388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:00.846485 (+ 97us) server_negotiation.cc:407] Connection header received
0928 04:49:00.846579 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:00.846580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:00.846595 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:00.846670 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:00.846993 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.847529 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.848231 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:00.848425 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:00.848503 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:00.850789 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39887: BlockingRecv error: recv got EOF from 127.0.98.3:39887 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 04:49:01 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:01 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:01.580156 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:01.575986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:40907)
0928 04:49:01.576117 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:01.576128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:01.576180 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:01.576309 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:01.576310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:01.576499 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:01.576546 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:01.576548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.577163 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.577164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:01.577717 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:01.577719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.578004 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.578005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:01.578040 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:01.578439 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:01.578443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:01.580054 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
W20260928 04:49:01.580196 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:01.576071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40907 (local address 127.0.98.62:41355)
0928 04:49:01.576189 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:01.576191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:01.576198 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:49:01.576217 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:01.576218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:01.576227 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:01.576301 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:01.576614 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.577097 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.577777 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.577945 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.577993 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:01.580129 (+ 2136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40907: BlockingRecv error: recv got EOF from 127.0.98.1:40907 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
Sep 28 04:49:01 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:01 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:01.734529 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:01.730020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:40187)
0928 04:49:01.730162 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:01.730173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:01.730266 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:01.730440 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:01.730441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:01.730611 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:01.730656 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:01.730674 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.731415 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.731417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:01.731943 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:01.731945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.732284 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.732285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:01.732339 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:01.732848 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:01.732852 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:01.734472 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
W20260928 04:49:01.734637 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:01.730159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40187 (local address 127.0.98.62:41355)
0928 04:49:01.730227 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:01.730230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:01.730239 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:49:01.730340 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:01.730341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:01.730353 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:01.730434 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:01.730738 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.731343 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.732024 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.732216 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.732288 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:01.734559 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40187: BlockingRecv error: recv got EOF from 127.0.98.2:40187 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 04:49:01 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:01 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:01.855878 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:01.851363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:51413)
0928 04:49:01.851581 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:01.851592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:01.851676 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:01.851818 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:01.851819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:01.851999 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:01.852042 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:01.852044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.852765 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.852766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:01.853351 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:01.853353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.853724 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.853725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:01.853774 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:01.854164 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:01.854168 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:01.855774 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
W20260928 04:49:01.855911 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:01.851416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51413 (local address 127.0.98.62:41355)
0928 04:49:01.851476 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:01.851478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:01.851657 (+ 179us) server_negotiation.cc:407] Connection header received
0928 04:49:01.851744 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:01.851745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:01.851757 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:01.851802 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:01.852143 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.852661 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.853435 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:01.853630 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:01.853722 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:01.855851 (+ 2129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51413: BlockingRecv error: recv got EOF from 127.0.98.3:51413 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 04:49:02 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:02 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:02.585537 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:02.580681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:59061)
0928 04:49:02.580794 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:02.580806 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:02.580868 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:02.581254 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:02.581256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:02.581441 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:02.581487 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:02.581489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.582112 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.582112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:02.582643 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:02.582644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.583140 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.583141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:02.583205 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:02.583651 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:02.583655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:02.585444 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":81,"threads_started":1}
W20260928 04:49:02.585582 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:02.580763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59061 (local address 127.0.98.62:41355)
0928 04:49:02.581015 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:02.581017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:02.581027 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:49:02.581059 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:02.581060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:02.581072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:02.581242 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:02.581561 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.582043 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.582718 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.583059 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.583128 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:02.585515 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59061: BlockingRecv error: recv got EOF from 127.0.98.1:59061 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":201,"threads_started":1}
Sep 28 04:49:02 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:02 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:02.739317 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:02.734990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:41641)
0928 04:49:02.735129 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:02.735139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:02.735200 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:02.735312 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:02.735313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:02.735484 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:02.735527 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:02.735529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.736150 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.736151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:02.736660 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:02.736661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.736974 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.736975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:02.737011 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:02.737510 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:02.737515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:02.739217 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
W20260928 04:49:02.739356 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:02.735043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41641 (local address 127.0.98.62:41355)
0928 04:49:02.735132 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:02.735134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:02.735195 (+ 61us) server_negotiation.cc:407] Connection header received
0928 04:49:02.735241 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:02.735241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:02.735250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:02.735295 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:02.735601 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.736087 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.736741 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.736915 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.736969 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:02.739297 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41641: BlockingRecv error: recv got EOF from 127.0.98.2:41641 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 04:49:02 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:02 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:02.860880 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:02.856415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:57753)
0928 04:49:02.856605 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:02.856617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:02.856693 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:02.856854 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:02.856855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:02.857030 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:02.857076 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:02.857077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.857759 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.857760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:02.858303 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:02.858304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.858704 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.858705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:02.858764 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:02.859200 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:02.859204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:02.860788 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
W20260928 04:49:02.860935 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:02.856502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57753 (local address 127.0.98.62:41355)
0928 04:49:02.856618 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:02.856620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:02.856685 (+ 65us) server_negotiation.cc:407] Connection header received
0928 04:49:02.856766 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:02.856767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:02.856782 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:02.856847 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:02.857159 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.857673 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.858390 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:02.858607 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:02.858703 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:02.860868 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57753: BlockingRecv error: recv got EOF from 127.0.98.3:57753 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 04:49:03 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:03 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:03.590375 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:03.586067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:36059)
0928 04:49:03.586209 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:03.586218 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:03.586262 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:03.586383 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:03.586384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:03.586524 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:03.586571 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:03.586572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.587213 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.587214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:03.587736 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:03.587738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.588094 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.588095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:03.588132 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:03.588722 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:03.588726 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:03.590321 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
W20260928 04:49:03.590547 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:03.586141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36059 (local address 127.0.98.62:41355)
0928 04:49:03.586261 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:03.586262 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:03.586270 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:49:03.586305 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:03.586306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:03.586314 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:03.586370 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:03.586627 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.587147 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.587829 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.588024 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.588087 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:03.590446 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36059: BlockingRecv error: recv got EOF from 127.0.98.1:36059 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
Sep 28 04:49:03 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:03 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:03.744511 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:03.739942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:58837)
0928 04:49:03.740106 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:03.740126 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:03.740194 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:03.740322 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:03.740323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:03.740482 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:03.740545 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:03.740547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.741132 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.741134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:03.741628 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:03.741629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.742145 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.742146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:03.742211 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:03.742711 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:03.742715 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:03.744452 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
W20260928 04:49:03.744637 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:03.740060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58837 (local address 127.0.98.62:41355)
0928 04:49:03.740129 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:03.740131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:03.740177 (+ 46us) server_negotiation.cc:407] Connection header received
0928 04:49:03.740242 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:03.740243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:03.740252 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:03.740301 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:03.740609 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.741048 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.741711 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.742059 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.742124 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:03.744559 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58837: BlockingRecv error: recv got EOF from 127.0.98.2:58837 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 04:49:03 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:03 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:03.865739 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:03.861455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:33747)
0928 04:49:03.861644 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:03.861654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:03.861704 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:03.861836 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:03.861837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:03.861997 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:03.862041 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:03.862042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.862659 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.862660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:03.863160 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:03.863161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.863454 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.863455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:03.863491 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:03.863940 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:03.863944 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:03.865674 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
W20260928 04:49:03.865803 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:03.861537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33747 (local address 127.0.98.62:41355)
0928 04:49:03.861599 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:03.861601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:03.861706 (+ 105us) server_negotiation.cc:407] Connection header received
0928 04:49:03.861759 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:03.861760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:03.861772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:03.861813 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:03.862105 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.862587 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.863236 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:03.863401 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:03.863443 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:03.865756 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33747: BlockingRecv error: recv got EOF from 127.0.98.3:33747 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 04:49:04 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:04 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:04.594733 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:04.590754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:56099)
0928 04:49:04.590913 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:04.590924 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:04.590971 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:04.591084 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:04.591085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:04.591244 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:04.591289 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:04.591290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.591878 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.591879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:04.592385 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:04.592387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.592663 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.592664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:04.592696 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:04.593096 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:04.593100 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:04.594677 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
W20260928 04:49:04.594841 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:04.590865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56099 (local address 127.0.98.62:41355)
0928 04:49:04.590978 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:04.590980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:04.590988 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:49:04.591008 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:04.591009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:04.591018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:04.591071 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:04.591348 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.591819 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.592445 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.592609 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.592651 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:04.594756 (+ 2105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56099: BlockingRecv error: recv got EOF from 127.0.98.1:56099 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
Sep 28 04:49:04 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:04 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:04.749673 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:04.744948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:53831)
0928 04:49:04.745124 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:04.745134 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:04.745249 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:04.745378 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:04.745379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:04.745541 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:04.745584 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:04.745585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.746333 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.746334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:04.746840 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:04.746842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.747220 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.747221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:04.747278 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:04.747732 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:04.747736 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:04.749618 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
W20260928 04:49:04.749804 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:04.745026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53831 (local address 127.0.98.62:41355)
0928 04:49:04.745118 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:04.745120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:04.745230 (+ 110us) server_negotiation.cc:407] Connection header received
0928 04:49:04.745306 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:04.745307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:04.745319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:04.745362 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:04.745676 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.746254 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.746924 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.747127 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.747225 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:04.749700 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53831: BlockingRecv error: recv got EOF from 127.0.98.2:53831 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 04:49:04 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:04 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:04.870568 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:04.866171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:59625)
0928 04:49:04.866328 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:04.866339 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:04.866400 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:04.866532 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:04.866533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:04.866704 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:04.866748 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:04.866749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.867422 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.867423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:04.867968 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:04.867969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.868331 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.868332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:04.868392 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:04.868864 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:04.868868 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:04.870481 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
W20260928 04:49:04.870610 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:04.866247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59625 (local address 127.0.98.62:41355)
0928 04:49:04.866317 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:04.866319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:04.866393 (+ 74us) server_negotiation.cc:407] Connection header received
0928 04:49:04.866450 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:04.866451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:04.866465 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:04.866511 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:04.866831 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.867335 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.868057 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:04.868250 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:04.868322 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:04.870551 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59625: BlockingRecv error: recv got EOF from 127.0.98.3:59625 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 04:49:05 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:05 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:05.599305 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:05.595222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:38253)
0928 04:49:05.595335 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:05.595344 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:05.595393 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:05.595515 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:05.595516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:05.595656 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:05.595701 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:05.595702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.596269 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.596270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:05.596766 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:05.596768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.597058 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.597058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:05.597092 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:05.597536 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:05.597540 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:05.599248 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
W20260928 04:49:05.599445 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:05.595306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38253 (local address 127.0.98.62:41355)
0928 04:49:05.595392 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:05.595394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:05.595403 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:49:05.595445 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:05.595446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:05.595455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:05.595501 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:05.595752 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.596207 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.596830 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.596998 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.597047 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:05.599359 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38253: BlockingRecv error: recv got EOF from 127.0.98.1:38253 (error 108)
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":47,"threads_started":1}
Sep 28 04:49:05 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:05 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:05.758291 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:05.750152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:39263)
0928 04:49:05.750362 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:05.750387 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:05.750473 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:05.750614 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:05.750614 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:05.750778 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:05.750821 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:05.750823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.751478 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.751479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:05.752018 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:05.752020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.752388 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.752390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:05.752452 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:05.756385 (+ 3933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:05.756390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:05.758232 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
W20260928 04:49:05.758397 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:05.750240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39263 (local address 127.0.98.62:41355)
0928 04:49:05.750307 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:05.750309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:05.750464 (+ 155us) server_negotiation.cc:407] Connection header received
0928 04:49:05.750527 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:05.750528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:05.750542 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:05.750609 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:05.750899 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.751385 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.752099 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.752301 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.752388 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:05.758327 (+ 5939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39263: BlockingRecv error: recv got EOF from 127.0.98.2:39263 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 04:49:05 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:05 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:05.875017 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:05.871039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:58343)
0928 04:49:05.871194 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:05.871212 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:05.871266 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:05.871408 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:05.871409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:05.871559 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:05.871604 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:05.871605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.872158 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.872159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:05.872636 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:05.872637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.872909 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.872909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:05.872941 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:05.873383 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:05.873387 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:05.874955 (+ 1568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
W20260928 04:49:05.875138 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:05.871113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58343 (local address 127.0.98.62:41355)
0928 04:49:05.871183 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:05.871185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:05.871256 (+ 71us) server_negotiation.cc:407] Connection header received
0928 04:49:05.871306 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:05.871307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:05.871317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:05.871377 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:05.871663 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.872100 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.872699 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:05.872856 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:05.872898 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:05.875063 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58343: BlockingRecv error: recv got EOF from 127.0.98.3:58343 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 04:49:06 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:06 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:06.604550 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:06.599868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:38627)
0928 04:49:06.600000 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:06.600015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:06.600062 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:06.600196 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:06.600196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:06.600343 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:06.600387 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:06.600389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.601008 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.601009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:06.601611 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:06.601613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.601927 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.601928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:06.601982 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:06.602487 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:06.602492 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:06.604456 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":92,"threads_started":1}
W20260928 04:49:06.604599 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:06.599944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38627 (local address 127.0.98.62:41355)
0928 04:49:06.600068 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:06.600069 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:06.600079 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:49:06.600101 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:06.600101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:06.600111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:06.600175 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:06.600443 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.600912 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.601675 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.601859 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.601908 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:06.604531 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38627: BlockingRecv error: recv got EOF from 127.0.98.1:38627 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
Sep 28 04:49:06 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:06 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:06.763028 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:06.758894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:48225)
0928 04:49:06.759038 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:06.759048 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:06.759107 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:06.759235 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:06.759236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:06.759389 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:06.759437 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:06.759438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.760018 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.760018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:06.760481 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:06.760482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.760772 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.760772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:06.760806 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:06.761307 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:06.761311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:06.762969 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
W20260928 04:49:06.763120 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:06.758954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48225 (local address 127.0.98.62:41355)
0928 04:49:06.759026 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:06.759028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:06.759110 (+ 82us) server_negotiation.cc:407] Connection header received
0928 04:49:06.759152 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:06.759153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:06.759164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:06.759211 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:06.759511 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.759957 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.760546 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.760715 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.760759 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:06.763054 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48225: BlockingRecv error: recv got EOF from 127.0.98.2:48225 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 04:49:06 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:06 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:06.879662 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:06.875434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:53055)
0928 04:49:06.875558 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:06.875572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:06.875630 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:06.875749 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:06.875750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:06.875894 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:06.875937 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:06.875939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.876544 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.876544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:06.877064 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:06.877065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.877486 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.877491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:06.877532 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:06.877993 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:06.877997 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:06.879606 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":53,"threads_started":1}
W20260928 04:49:06.879824 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:06.875564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53055 (local address 127.0.98.62:41355)
0928 04:49:06.875636 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:06.875638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:06.875665 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:49:06.875685 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:06.875685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:06.875697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:06.875735 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:06.875997 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.876482 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.877136 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:06.877426 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:06.877483 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:06.879747 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53055: BlockingRecv error: recv got EOF from 127.0.98.3:53055 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 04:49:07 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:07 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:07.609495 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:07.605020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:38829)
0928 04:49:07.605136 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:07.605149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:07.605248 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:07.605394 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:07.605394 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:07.605565 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:07.605613 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:07.605614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.606225 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.606226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:07.606742 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:07.606743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.607054 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.607055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:07.607092 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:07.607591 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:07.607595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:07.609402 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
W20260928 04:49:07.609570 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:07.605081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38829 (local address 127.0.98.62:41355)
0928 04:49:07.605222 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:07.605224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:07.605233 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:49:07.605297 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:07.605298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:07.605307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:07.605363 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:07.605683 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.606164 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.606812 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.606995 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.607042 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:07.609495 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38829: BlockingRecv error: recv got EOF from 127.0.98.1:38829 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
Sep 28 04:49:07 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:07 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:07.768055 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:07.763583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:52127)
0928 04:49:07.763719 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:07.763731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:07.763795 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:07.763921 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:07.763922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:07.764075 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:07.764122 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:07.764123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.764745 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.764746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:07.765344 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:07.765346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.765661 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.765661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:07.765714 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:07.766227 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:07.766231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:07.767982 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
W20260928 04:49:07.768162 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:07.763647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52127 (local address 127.0.98.62:41355)
0928 04:49:07.763716 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:07.763718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:07.763798 (+ 80us) server_negotiation.cc:407] Connection header received
0928 04:49:07.763837 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:07.763838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:07.763848 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:07.763899 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:07.764185 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.764680 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.765415 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.765592 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.765650 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:07.768085 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52127: BlockingRecv error: recv got EOF from 127.0.98.2:52127 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 04:49:07 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:07 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:07.884857 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:07.880263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:59197)
0928 04:49:07.880456 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:07.880468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:07.880559 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:07.880708 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:07.880709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:07.880888 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:07.880932 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:07.880934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.881650 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.881652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:07.882182 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:07.882184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.882564 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.882565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:07.882626 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:07.883088 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:07.883092 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:07.884798 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":104,"threads_started":1}
W20260928 04:49:07.885012 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:07.880376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59197 (local address 127.0.98.62:41355)
0928 04:49:07.880450 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:07.880453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:07.880540 (+ 87us) server_negotiation.cc:407] Connection header received
0928 04:49:07.880613 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:07.880614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:07.880628 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:07.880702 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:07.881011 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.881570 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.882274 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:07.882460 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:07.882565 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:07.884880 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59197: BlockingRecv error: recv got EOF from 127.0.98.3:59197 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 04:49:08 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:08 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:08.614789 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:08.609968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:47041)
0928 04:49:08.610134 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:08.610145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:08.610206 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:08.610344 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:08.610345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:08.610506 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:08.610552 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:08.610553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.611214 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.611215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:08.611784 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:08.611786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.612103 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.612103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:08.612143 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:08.612616 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:08.612620 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:08.614674 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
W20260928 04:49:08.614862 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:08.610073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47041 (local address 127.0.98.62:41355)
0928 04:49:08.610203 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:08.610205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:08.610214 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:49:08.610255 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:08.610255 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:08.610266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:08.610324 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:08.610613 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.611137 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.611859 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.612042 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.612091 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:08.614772 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47041: BlockingRecv error: recv got EOF from 127.0.98.1:47041 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
Sep 28 04:49:08 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:08 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:08.773339 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:08.768595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:56497)
0928 04:49:08.768744 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:08.768760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:08.768809 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:08.768962 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:08.768962 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:08.769118 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:08.769163 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:08.769164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.770101 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.770102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:08.770625 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:08.770626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.770936 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.770937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:08.770971 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:08.771499 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:08.771503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:08.773280 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
W20260928 04:49:08.773515 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:08.768707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56497 (local address 127.0.98.62:41355)
0928 04:49:08.768775 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:08.768777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:08.768795 (+ 18us) server_negotiation.cc:407] Connection header received
0928 04:49:08.768871 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:08.768872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:08.768882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:08.768942 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:08.769292 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.770035 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.770702 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.770881 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.770930 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:08.773430 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56497: BlockingRecv error: recv got EOF from 127.0.98.2:56497 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 04:49:08 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:08 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:08.890297 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:08.885454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:43059)
0928 04:49:08.885587 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:08.885600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:08.885668 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:08.885818 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:08.885819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:08.885980 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:08.886025 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:08.886027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.886671 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.886672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:08.887249 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:08.887251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.887734 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.887735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:08.887792 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:08.888323 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:08.888328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:08.890226 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260928 04:49:08.890358 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:08.885552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43059 (local address 127.0.98.62:41355)
0928 04:49:08.885653 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:08.885654 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:08.885665 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:49:08.885735 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:08.885735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:08.885748 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:08.885798 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:08.886093 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.886599 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.887328 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:08.887656 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:08.887724 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:08.890311 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43059: BlockingRecv error: recv got EOF from 127.0.98.3:43059 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 04:49:09 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:09 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:09.620334 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:09.615398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:44447)
0928 04:49:09.615533 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:09.615548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:09.615675 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:09.615873 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:09.615874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:09.616036 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:09.616087 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:09.616089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.616760 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.616761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:09.617307 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:09.617308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.617775 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.617776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:09.617836 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:09.618330 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:09.618335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:09.620275 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260928 04:49:09.620509 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:09.615505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44447 (local address 127.0.98.62:41355)
0928 04:49:09.615651 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:09.615653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:09.615662 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:49:09.615770 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:09.615771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:09.615782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:09.615849 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:09.616156 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.616661 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.617385 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.617633 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.617689 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:09.620418 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44447: BlockingRecv error: recv got EOF from 127.0.98.1:44447 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
Sep 28 04:49:09 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:09 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:09.778268 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:09.773815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:54453)
0928 04:49:09.773971 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:09.773984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:09.774043 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:09.774176 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:09.774176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:09.774329 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:09.774376 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:09.774378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.775009 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.775009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:09.775527 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:09.775528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.775837 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.775837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:09.775874 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:09.776337 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:09.776341 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:09.778143 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
W20260928 04:49:09.778306 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:09.773952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54453 (local address 127.0.98.62:41355)
0928 04:49:09.774015 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:09.774018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:09.774041 (+ 23us) server_negotiation.cc:407] Connection header received
0928 04:49:09.774101 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:09.774102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:09.774112 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:09.774154 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:09.774443 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.774934 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.775600 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.775776 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.775826 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:09.778236 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54453: BlockingRecv error: recv got EOF from 127.0.98.2:54453 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 04:49:09 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:09 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:09.895725 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:09.890867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:48137)
0928 04:49:09.891029 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:09.891039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:09.891096 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:09.891268 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:09.891270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:09.891458 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:09.891508 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:09.891510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.892199 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.892201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:09.892732 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:09.892733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.893130 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.893131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:09.893219 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:09.893760 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:09.893765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:09.895664 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
W20260928 04:49:09.895879 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:09.890899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48137 (local address 127.0.98.62:41355)
0928 04:49:09.890953 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:09.890956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:09.891085 (+ 129us) server_negotiation.cc:407] Connection header received
0928 04:49:09.891164 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:09.891164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:09.891179 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:09.891257 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:09.891599 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.892107 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.892826 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:09.893040 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:09.893129 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:09.895772 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48137: BlockingRecv error: recv got EOF from 127.0.98.3:48137 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 04:49:10 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:10 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:10.625584 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:10.620854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:44741)
0928 04:49:10.620974 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:10.620988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:10.621071 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:10.621295 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:10.621297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:10.621529 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:10.621614 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:10.621617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.622376 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.622377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:10.622894 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:10.622895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.623208 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.623209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:10.623247 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:10.623672 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:10.623676 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:10.625474 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
W20260928 04:49:10.625626 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:10.620942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44741 (local address 127.0.98.62:41355)
0928 04:49:10.621078 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:10.621080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:10.621108 (+ 28us) server_negotiation.cc:407] Connection header received
0928 04:49:10.621131 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:10.621132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:10.621143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:10.621286 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:10.621699 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.622308 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.622963 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.623148 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.623197 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:10.625553 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44741: BlockingRecv error: recv got EOF from 127.0.98.1:44741 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
Sep 28 04:49:10 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:10 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:10.789144 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:10.778779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:56571)
0928 04:49:10.778912 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:10.778922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:10.779001 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:10.779154 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:10.779155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:10.779331 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:10.779376 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:10.779377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.780061 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.780063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:10.780590 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:10.780591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.780973 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.780974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:10.781030 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:10.787144 (+ 6114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:10.787149 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:10.789064 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":55,"threads_started":1}
W20260928 04:49:10.789274 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:10.778914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56571 (local address 127.0.98.62:41355)
0928 04:49:10.779005 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:10.779007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:10.779017 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:49:10.779054 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:10.779055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:10.779068 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:10.779149 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:10.779461 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.779961 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.780682 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.780875 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.780973 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:10.789163 (+ 8190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56571: BlockingRecv error: recv got EOF from 127.0.98.2:56571 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 04:49:10 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:10 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:10.900594 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:10.896232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.3:57139)
0928 04:49:10.896426 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:10.896437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:10.896533 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:10.896682 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:10.896683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:10.896838 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:10.896886 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:10.896887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.897559 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.897559 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:10.898069 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:10.898070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.898426 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.898427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:10.898463 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:10.898915 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:10.898919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:10.900527 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
W20260928 04:49:10.900677 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:10.896286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57139 (local address 127.0.98.62:41355)
0928 04:49:10.896370 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:10.896372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:10.896536 (+ 164us) server_negotiation.cc:407] Connection header received
0928 04:49:10.896603 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:10.896604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:10.896617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:10.896660 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:10.896949 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.897492 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.898166 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:10.898356 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:10.898426 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:10.900627 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57139: BlockingRecv error: recv got EOF from 127.0.98.3:57139 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 04:49:11 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:11 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:11.630924 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:11.626138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.1:57331)
0928 04:49:11.626294 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:11.626305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:11.626358 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:11.626479 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:11.626480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:11.626642 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:11.626690 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:11.626692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:11.627340 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:11.627341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:11.627867 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:11.627868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:11.628250 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:11.628250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:11.628311 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:11.628832 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:11.628837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:11.630817 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
W20260928 04:49:11.630975 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:11.626243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57331 (local address 127.0.98.62:41355)
0928 04:49:11.626358 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:11.626360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:11.626369 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:49:11.626395 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:11.626395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:11.626406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:11.626465 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:11.626757 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:11.627247 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:11.627959 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:11.628167 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:11.628251 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:11.630899 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57331: BlockingRecv error: recv got EOF from 127.0.98.1:57331 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
Sep 28 04:49:11 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 04:49:11 dist-test-slave-jccs krb5kdc[8538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 04:49:11.794847 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:11.789704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41355 (local address 127.0.98.2:37905)
0928 04:49:11.789968 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:11.789980 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:11.790066 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:11.790240 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:11.790240 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:11.790409 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:11.790453 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:11.790455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:11.791136 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:11.791137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:11.791841 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:11.791843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:11.792262 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:11.792263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:11.792317 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:11.792821 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:11.792826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:11.794781 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41355: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":187,"threads_started":1}
W20260928 04:49:11.794966 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:11.789883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37905 (local address 127.0.98.62:41355)
0928 04:49:11.789956 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:11.789959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:11.790046 (+ 87us) server_negotiation.cc:407] Connection header received
0928 04:49:11.790149 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:11.790150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:11.790161 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:11.790218 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:11.790538 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:11.791040 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:11.791945 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:11.792172 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:11.792262 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:11.794870 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37905: BlockingRecv error: recv got EOF from 127.0.98.2:37905 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:49:11.828111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 9121
I20260928 04:49:11.832576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 9273
I20260928 04:49:11.836856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 9406
I20260928 04:49:11.841050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 9051
2026-09-28T04:49:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17403 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T04:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:49:11Z Disabled control of system clock
I20260928 04:49:11.868039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34413
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 04:49:11.947870 9601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:11.948097 9601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:11.948119 9601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:11.949402 9601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:49:11.949450 9601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:11.949463 9601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:11.949476 9601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:49:11.949487 9601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:49:11.950907 9601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:11.951190 9601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:11.951501 9601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:49:11.956722 9601 server_base.cc:1034] running on GCE node
W20260928 04:49:11.956815 9609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:11.956902 9606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:11.956902 9607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:11.957304 9601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:11.957652 9601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:11.958803 9601 hybrid_clock.cc:648] HybridClock initialized: now 1790570951958788 us; error 33 us; skew 500 ppm
I20260928 04:49:11.960161 9601 webserver.cc:460] Webserver started at http://127.0.98.62:39909/ using document root <none> and password file <none>
I20260928 04:49:11.960426 9601 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:11.960501 9601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:11.960659 9601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:11.961860 9601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "1e496b5117b140638a10449a04083949"
format_stamp: "Formatted at 2026-09-28 04:49:11 on dist-test-slave-jccs"
server_key: "fb4b6c11fd55c2230f926291355e61cb"
server_key_iv: "569a43849efd3209b182cd9e3d3bff91"
server_key_version: "encryptionkey@0"
I20260928 04:49:11.962239 9601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e496b5117b140638a10449a04083949"
format_stamp: "Formatted at 2026-09-28 04:49:11 on dist-test-slave-jccs"
server_key: "fb4b6c11fd55c2230f926291355e61cb"
server_key_iv: "569a43849efd3209b182cd9e3d3bff91"
server_key_version: "encryptionkey@0"
I20260928 04:49:11.963543 9601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:49:11.964326 9615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:11.964499 9601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:49:11.964622 9601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "1e496b5117b140638a10449a04083949"
format_stamp: "Formatted at 2026-09-28 04:49:11 on dist-test-slave-jccs"
server_key: "fb4b6c11fd55c2230f926291355e61cb"
server_key_iv: "569a43849efd3209b182cd9e3d3bff91"
server_key_version: "encryptionkey@0"
I20260928 04:49:11.964704 9601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:11.971334 9601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:11.971669 9601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:11.971830 9601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:11.975803 9601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34413
I20260928 04:49:11.975847 9667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34413 every 8 connection(s)
I20260928 04:49:11.976213 9601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:49:11.976948 9668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:49:11.979698 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949: Bootstrap starting.
I20260928 04:49:11.980242 9668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949: Neither blocks nor log segments found. Creating new log.
I20260928 04:49:11.980468 9668 log.cc:824] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949: Log is configured to *not* fsync() on all Append() calls
I20260928 04:49:11.981081 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949: No bootstrap required, opened a new log
I20260928 04:49:11.982237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 9601
I20260928 04:49:11.982342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:49:11.982553 392 external_mini_cluster.cc:1428] Setting key d161463bd77fe80925b848bb1f744be1
I20260928 04:49:11.982870 9668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e496b5117b140638a10449a04083949" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34413 } }
I20260928 04:49:11.983088 9668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:49:11.983134 9668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e496b5117b140638a10449a04083949, State: Initialized, Role: FOLLOWER
I20260928 04:49:11.983295 9668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e496b5117b140638a10449a04083949" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34413 } }
I20260928 04:49:11.983392 9668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:49:11.983426 9668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:49:11.983472 9668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:49:11.984182 9668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e496b5117b140638a10449a04083949" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34413 } }
I20260928 04:49:11.984288 9668 leader_election.cc:302] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e496b5117b140638a10449a04083949; no voters:
I20260928 04:49:11.984423 9668 leader_election.cc:288] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:49:11.984660 9673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:49:11.984678 9668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:49:11.984850 9673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [term 1 LEADER]: Becoming Leader. State: Replica: 1e496b5117b140638a10449a04083949, State: Running, Role: LEADER
I20260928 04:49:11.984938 9673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e496b5117b140638a10449a04083949" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34413 } }
I20260928 04:49:11.985363 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e496b5117b140638a10449a04083949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e496b5117b140638a10449a04083949" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34413 } } }
I20260928 04:49:11.985389 9675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e496b5117b140638a10449a04083949. Latest consensus state: current_term: 1 leader_uuid: "1e496b5117b140638a10449a04083949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e496b5117b140638a10449a04083949" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34413 } } }
I20260928 04:49:11.985457 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [sys.catalog]: This master's current role is: LEADER
I20260928 04:49:11.985464 9675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949 [sys.catalog]: This master's current role is: LEADER
I20260928 04:49:11.986384 9682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:49:11.986965 9682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:49:11.988706 9682 catalog_manager.cc:1347] Generated new cluster ID: ef56362163134a078ddc3396d1cd7a5a
I20260928 04:49:11.988739 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:11.983123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.0.98.62:34413)
0928 04:49:11.983388 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:11.983398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:11.983431 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:49:11.983547 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:11.983553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:11.983625 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:11.983767 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:11.983977 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:11.985307 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:11.986455 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:11.987036 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:11.987198 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:11.987959 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:11.987963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:11.987964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:49:11.988177 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:11.988241 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:49:11.988257 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:11.988329 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:11.988435 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 04:49:11.988510 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260928 04:49:11.988762 9682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:49:12.004362 9682 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:49:12.005105 9682 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:49:12.009481 9682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e496b5117b140638a10449a04083949: Generated new TSK 0
I20260928 04:49:12.009684 9682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:49:12.013832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34413
--builtin_ntp_servers=127.0.98.20:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 04:49:12.094713 9692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:12.094966 9692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:12.095002 9692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:12.096207 9692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:12.096248 9692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:12.096273 9692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:49:12.097625 9692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:12.097858 9692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:12.098160 9692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:12.103931 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.103937 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.103955 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:12.104245 9692 server_base.cc:1034] running on GCE node
I20260928 04:49:12.104458 9692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:12.104733 9692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:12.105886 9692 hybrid_clock.cc:648] HybridClock initialized: now 1790570952105864 us; error 39 us; skew 500 ppm
I20260928 04:49:12.107300 9692 webserver.cc:460] Webserver started at http://127.0.98.1:36719/ using document root <none> and password file <none>
I20260928 04:49:12.107538 9692 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:12.107583 9692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:12.107738 9692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:12.108866 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "2648b31e6161464982dec65baecd6457"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "d89e8936fb31eff6b5c3d6d7a7b53d47"
server_key_iv: "17169cc45ae436ebd3c20c7771810618"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.109275 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2648b31e6161464982dec65baecd6457"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "d89e8936fb31eff6b5c3d6d7a7b53d47"
server_key_iv: "17169cc45ae436ebd3c20c7771810618"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.110606 9692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:49:12.111377 9706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.111528 9692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:49:12.111632 9692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "2648b31e6161464982dec65baecd6457"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "d89e8936fb31eff6b5c3d6d7a7b53d47"
server_key_iv: "17169cc45ae436ebd3c20c7771810618"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.111712 9692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:12.127368 9692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:12.127681 9692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:12.127839 9692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:12.128068 9692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:12.128484 9692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:12.128535 9692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.128597 9692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:12.128628 9692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.134505 9692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38411
I20260928 04:49:12.134548 9819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38411 every 8 connection(s)
I20260928 04:49:12.134879 9692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:49:12.138942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 9692
I20260928 04:49:12.139043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:49:12.139247 392 external_mini_cluster.cc:1428] Setting key f2b4a31cd11bc5dc9fe9fcfd8d9f176d
I20260928 04:49:12.140429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34413
--builtin_ntp_servers=127.0.98.20:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 04:49:12.140540 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.135608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55143 (local address 127.0.98.62:34413)
0928 04:49:12.135680 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:12.135683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:12.135953 (+ 270us) server_negotiation.cc:407] Connection header received
0928 04:49:12.136446 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:12.136447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:12.136458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:12.136521 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:12.137164 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.137727 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.138830 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.139150 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.139243 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:12.140242 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:12.140243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:12.140243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:49:12.140248 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:12.140265 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:49:12.140270 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:12.140331 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:12.140407 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 04:49:12.140443 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:49:12.140797 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.135588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34413 (local address 127.0.98.1:55143)
0928 04:49:12.135809 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:12.135853 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:12.136358 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:12.136592 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:12.136595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:12.136604 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:12.137079 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:12.137083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.137904 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.137905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:12.138728 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:12.138730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.139260 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.139261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:12.139443 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:12.140096 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:49:12.140097 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:12.140126 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:49:12.140150 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:12.140336 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:12.140340 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:12.140343 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:49:12.140392 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:49:12.140471 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":116,"threads_started":1}
I20260928 04:49:12.141327 9820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34413
I20260928 04:49:12.141433 9820 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:12.141629 9820 heartbeater.cc:511] Master 127.0.98.62:34413 requested a full tablet report, sending...
I20260928 04:49:12.142177 9632 ts_manager.cc:194] Registered new tserver with Master: 2648b31e6161464982dec65baecd6457 (127.0.98.1:38411)
I20260928 04:49:12.143107 9632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55143
W20260928 04:49:12.222398 9823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:12.222688 9823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:12.222718 9823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:12.223855 9823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:12.223891 9823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:12.223914 9823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:49:12.225244 9823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:12.225442 9823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:12.225680 9823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:12.230831 9831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.230854 9828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.230795 9829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:12.230935 9823 server_base.cc:1034] running on GCE node
I20260928 04:49:12.231165 9823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:12.231422 9823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:12.232606 9823 hybrid_clock.cc:648] HybridClock initialized: now 1790570952232601 us; error 40 us; skew 500 ppm
I20260928 04:49:12.233906 9823 webserver.cc:460] Webserver started at http://127.0.98.2:42977/ using document root <none> and password file <none>
I20260928 04:49:12.234143 9823 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:12.234221 9823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:12.234385 9823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:12.235533 9823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b2a1e9144b342ba8aad026c2abedd8d"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "c5c5352d6e4f50c5554e712c8d87955d"
server_key_iv: "f18d150070480f8c975d23c3b709f85c"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.235905 9823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b2a1e9144b342ba8aad026c2abedd8d"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "c5c5352d6e4f50c5554e712c8d87955d"
server_key_iv: "f18d150070480f8c975d23c3b709f85c"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.237246 9823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:49:12.237993 9837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.238171 9823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:49:12.238276 9823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "1b2a1e9144b342ba8aad026c2abedd8d"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "c5c5352d6e4f50c5554e712c8d87955d"
server_key_iv: "f18d150070480f8c975d23c3b709f85c"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.238363 9823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:12.253768 9823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:12.254098 9823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:12.254257 9823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:12.254489 9823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:12.254822 9823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:12.254876 9823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.254910 9823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:12.254937 9823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.260675 9823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40889
I20260928 04:49:12.260728 9950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40889 every 8 connection(s)
I20260928 04:49:12.261061 9823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:49:12.265626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 9823
I20260928 04:49:12.265729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:49:12.265892 392 external_mini_cluster.cc:1428] Setting key efef1f0744657aef7f645b06a7adbf77
I20260928 04:49:12.266399 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.261899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58263 (local address 127.0.98.62:34413)
0928 04:49:12.261950 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:12.261952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:12.262218 (+ 266us) server_negotiation.cc:407] Connection header received
0928 04:49:12.262681 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:12.262682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:12.262691 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:12.262750 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:12.263367 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.263882 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.264822 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.265001 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.265056 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:12.266040 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:12.266042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:12.266042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:49:12.266047 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:12.266063 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:49:12.266067 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:12.266128 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:12.266297 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 04:49:12.266321 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:49:12.266498 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.261904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34413 (local address 127.0.98.2:58263)
0928 04:49:12.262087 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:12.262123 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:12.262599 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:12.262804 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:12.262807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:12.262815 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:12.263298 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:12.263302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.263958 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.263959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:12.264743 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:12.264745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.265080 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.265081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:12.265243 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:12.265880 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:49:12.265880 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:12.265909 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:49:12.265932 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:12.266135 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:12.266141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:12.266144 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:49:12.266214 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:49:12.266312 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260928 04:49:12.266928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34413
--builtin_ntp_servers=127.0.98.20:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 04:49:12.267326 9951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34413
I20260928 04:49:12.267417 9951 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:12.267613 9951 heartbeater.cc:511] Master 127.0.98.62:34413 requested a full tablet report, sending...
I20260928 04:49:12.267956 9632 ts_manager.cc:194] Registered new tserver with Master: 1b2a1e9144b342ba8aad026c2abedd8d (127.0.98.2:40889)
I20260928 04:49:12.268352 9632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58263
W20260928 04:49:12.347743 9954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:12.347950 9954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:12.347970 9954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:12.349154 9954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:12.349273 9954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:12.349323 9954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:49:12.350657 9954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:12.350919 9954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:12.351240 9954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:12.356470 9959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.356453 9960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.356452 9962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:12.356556 9954 server_base.cc:1034] running on GCE node
I20260928 04:49:12.356936 9954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:12.357138 9954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:12.358314 9954 hybrid_clock.cc:648] HybridClock initialized: now 1790570952358305 us; error 27 us; skew 500 ppm
I20260928 04:49:12.359664 9954 webserver.cc:460] Webserver started at http://127.0.98.3:45729/ using document root <none> and password file <none>
I20260928 04:49:12.359946 9954 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:12.360001 9954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:12.360152 9954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:12.361373 9954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "e0c682167aff4ec390eeb549a5e9f97f"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "073153f40e67fe3f7d9a8f8bde96fffa"
server_key_iv: "a11106ff8527429d1b2ae258fea518f2"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.361799 9954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e0c682167aff4ec390eeb549a5e9f97f"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "073153f40e67fe3f7d9a8f8bde96fffa"
server_key_iv: "a11106ff8527429d1b2ae258fea518f2"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.363092 9954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:49:12.363780 9968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.363966 9954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:49:12.364065 9954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "e0c682167aff4ec390eeb549a5e9f97f"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "073153f40e67fe3f7d9a8f8bde96fffa"
server_key_iv: "a11106ff8527429d1b2ae258fea518f2"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.364143 9954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:12.390213 9954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:12.390548 9954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:12.390699 9954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:12.390933 9954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:12.391299 9954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:12.391359 9954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.391402 9954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:12.391429 9954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.397167 9954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39365
I20260928 04:49:12.397276 10081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39365 every 8 connection(s)
I20260928 04:49:12.397625 9954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:49:12.402129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 9954
I20260928 04:49:12.402209 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:49:12.402436 392 external_mini_cluster.cc:1428] Setting key 2d1b79de244dd41557b0a5a1f4bcd5d0
I20260928 04:49:12.402976 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.398403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55095 (local address 127.0.98.62:34413)
0928 04:49:12.398474 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:12.398476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:12.398697 (+ 221us) server_negotiation.cc:407] Connection header received
0928 04:49:12.399227 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:12.399228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:12.399239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:12.399302 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:12.399983 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.400485 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.401445 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.401634 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.401695 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:12.402608 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:12.402609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:12.402609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:49:12.402614 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:12.402633 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:49:12.402639 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:12.402700 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:12.402871 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 04:49:12.402900 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:49:12.403185 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.398387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34413 (local address 127.0.98.3:55095)
0928 04:49:12.398567 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:12.398602 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:12.399101 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:12.399376 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:12.399380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:12.399388 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:12.399886 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:12.399890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.400571 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.400572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:12.401374 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:12.401376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:12.401714 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:12.401714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:12.401816 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:12.402447 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:49:12.402448 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:12.402487 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:49:12.402516 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:12.402771 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:12.402775 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:12.402777 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:49:12.402858 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 04:49:12.402956 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260928 04:49:12.403628 10082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34413
I20260928 04:49:12.403716 10082 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:12.403899 10082 heartbeater.cc:511] Master 127.0.98.62:34413 requested a full tablet report, sending...
I20260928 04:49:12.404265 9632 ts_manager.cc:194] Registered new tserver with Master: e0c682167aff4ec390eeb549a5e9f97f (127.0.98.3:39365)
I20260928 04:49:12.404625 9632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55095
I20260928 04:49:12.414498 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 04:49:12.416457 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:12.415937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.0.98.62:34413)
0928 04:49:12.416002 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:12.416004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:12.416096 (+ 92us) server_negotiation.cc:407] Connection header received
0928 04:49:12.416143 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:12.416143 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:12.416155 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:12.416202 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:12.416362 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48500: BlockingRecv error: recv got EOF from 127.0.0.1:48500 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:49:12.416455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 9692
I20260928 04:49:12.420822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 9823
I20260928 04:49:12.424798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 9954
I20260928 04:49:12.429059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 9601
2026-09-28T04:49:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (587 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T04:49:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:49:12Z Disabled control of system clock
I20260928 04:49:12.456386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45387
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 04:49:12.536546 10093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:12.536765 10093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:12.536789 10093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:12.537988 10093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:49:12.538038 10093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:12.538051 10093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:12.538064 10093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:49:12.538077 10093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:49:12.539474 10093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:12.539744 10093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:12.540058 10093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:12.544960 10101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.545012 10099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.545053 10098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:12.545148 10093 server_base.cc:1034] running on GCE node
I20260928 04:49:12.545449 10093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:12.545763 10093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:12.546948 10093 hybrid_clock.cc:648] HybridClock initialized: now 1790570952546921 us; error 61 us; skew 500 ppm
I20260928 04:49:12.548166 10093 webserver.cc:460] Webserver started at http://127.0.98.62:34905/ using document root <none> and password file <none>
I20260928 04:49:12.548424 10093 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:12.548493 10093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:12.548655 10093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:12.549813 10093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "563a1080f7d442bfacc7f0d4223ce5e0"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "fca2befb20b4ec423720fdb0b5651f73"
server_key_iv: "c103634a0b12e8fe1ff901d75bbaca3f"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.550172 10093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "563a1080f7d442bfacc7f0d4223ce5e0"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "fca2befb20b4ec423720fdb0b5651f73"
server_key_iv: "c103634a0b12e8fe1ff901d75bbaca3f"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.551590 10093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:49:12.552327 10107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.552493 10093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:49:12.552592 10093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "563a1080f7d442bfacc7f0d4223ce5e0"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "fca2befb20b4ec423720fdb0b5651f73"
server_key_iv: "c103634a0b12e8fe1ff901d75bbaca3f"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.552675 10093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:12.556010 10093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:12.556269 10093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:12.556417 10093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:12.560159 10093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45387
I20260928 04:49:12.560187 10159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45387 every 8 connection(s)
I20260928 04:49:12.560504 10093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:49:12.560778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 10093
I20260928 04:49:12.560879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:49:12.561074 392 external_mini_cluster.cc:1428] Setting key d68894d10a9ec6681d0ad79a9f4f3559
I20260928 04:49:12.561295 10160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:49:12.564075 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.561925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.0.98.62:45387)
0928 04:49:12.562167 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:12.562174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:12.562199 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:49:12.562750 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:12.562756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:12.562794 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:12.562970 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:12.563664 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:12.563668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:12.563668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:49:12.563672 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:12.563720 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:49:12.563734 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:12.563814 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:12.563879 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 04:49:12.563934 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
I20260928 04:49:12.564459 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0: Bootstrap starting.
I20260928 04:49:12.565112 10160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0: Neither blocks nor log segments found. Creating new log.
I20260928 04:49:12.565428 10160 log.cc:824] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0: Log is configured to *not* fsync() on all Append() calls
I20260928 04:49:12.566130 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0: No bootstrap required, opened a new log
I20260928 04:49:12.567417 10160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563a1080f7d442bfacc7f0d4223ce5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45387 } }
I20260928 04:49:12.567576 10160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:49:12.567624 10160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563a1080f7d442bfacc7f0d4223ce5e0, State: Initialized, Role: FOLLOWER
I20260928 04:49:12.567739 10160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563a1080f7d442bfacc7f0d4223ce5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45387 } }
I20260928 04:49:12.567857 10160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:49:12.567905 10160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:49:12.567955 10160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:49:12.568442 10160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563a1080f7d442bfacc7f0d4223ce5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45387 } }
I20260928 04:49:12.568564 10160 leader_election.cc:302] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 563a1080f7d442bfacc7f0d4223ce5e0; no voters:
I20260928 04:49:12.568753 10160 leader_election.cc:288] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:49:12.568974 10165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:49:12.569013 10160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:49:12.569109 10165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 563a1080f7d442bfacc7f0d4223ce5e0, State: Running, Role: LEADER
I20260928 04:49:12.569271 10165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563a1080f7d442bfacc7f0d4223ce5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45387 } }
I20260928 04:49:12.569638 10166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "563a1080f7d442bfacc7f0d4223ce5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563a1080f7d442bfacc7f0d4223ce5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45387 } } }
I20260928 04:49:12.569658 10167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 563a1080f7d442bfacc7f0d4223ce5e0. Latest consensus state: current_term: 1 leader_uuid: "563a1080f7d442bfacc7f0d4223ce5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563a1080f7d442bfacc7f0d4223ce5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45387 } } }
I20260928 04:49:12.569734 10166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [sys.catalog]: This master's current role is: LEADER
I20260928 04:49:12.569747 10167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0 [sys.catalog]: This master's current role is: LEADER
I20260928 04:49:12.569959 10174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:49:12.570686 10174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:49:12.572103 10174 catalog_manager.cc:1347] Generated new cluster ID: 6e8f54379e134344a1d47e9b094e77e6
I20260928 04:49:12.572155 10174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:49:12.601763 10174 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:49:12.601980 10174 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:49:12.609987 10174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 563a1080f7d442bfacc7f0d4223ce5e0: Generated new TSK 0
I20260928 04:49:12.610216 10174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:49:12.616617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45387
--builtin_ntp_servers=127.0.98.20:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 04:49:12.701768 10184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:12.702633 10184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:12.702720 10184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:12.704450 10184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:12.704521 10184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:12.704573 10184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:49:12.706139 10184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:12.706405 10184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:12.706707 10184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:12.711826 10190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:12.711845 10184 server_base.cc:1034] running on GCE node
W20260928 04:49:12.711952 10192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.712057 10189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:12.712303 10184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:12.712541 10184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:12.713713 10184 hybrid_clock.cc:648] HybridClock initialized: now 1790570952713710 us; error 43 us; skew 500 ppm
I20260928 04:49:12.714908 10184 webserver.cc:460] Webserver started at http://127.0.98.1:42217/ using document root <none> and password file <none>
I20260928 04:49:12.715145 10184 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:12.715221 10184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:12.715373 10184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:12.716418 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "cefe5aafc5b14c0bbb912ac7e4fe5153"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "4f00ecb954b089bfa392fa72c3e2df0b"
server_key_iv: "f55369084926657eca301cf6b33b50de"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.716777 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cefe5aafc5b14c0bbb912ac7e4fe5153"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "4f00ecb954b089bfa392fa72c3e2df0b"
server_key_iv: "f55369084926657eca301cf6b33b50de"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.718120 10184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:49:12.718860 10198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.719033 10184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:49:12.719131 10184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "cefe5aafc5b14c0bbb912ac7e4fe5153"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "4f00ecb954b089bfa392fa72c3e2df0b"
server_key_iv: "f55369084926657eca301cf6b33b50de"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.719226 10184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:12.722587 10184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:12.722847 10184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:12.722991 10184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:12.723201 10184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:12.723497 10184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:12.723551 10184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.723588 10184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:12.723618 10184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.729137 10184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38535
I20260928 04:49:12.729229 10311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38535 every 8 connection(s)
I20260928 04:49:12.729547 10184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:49:12.732506 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.730285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59425 (local address 127.0.98.62:45387)
0928 04:49:12.730339 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:12.730341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:12.730545 (+ 204us) server_negotiation.cc:407] Connection header received
0928 04:49:12.731042 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:12.731043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:12.731051 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:12.731116 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:12.732246 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:12.732248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:12.732248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:49:12.732253 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:12.732268 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:49:12.732273 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:12.732332 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:12.732422 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 04:49:12.732437 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:49:12.732709 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.730230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45387 (local address 127.0.98.1:59425)
0928 04:49:12.730429 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:12.730463 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:12.730933 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:12.731212 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:12.731215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:12.731223 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:12.732079 (+ 856us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:49:12.732080 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:12.732106 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:49:12.732132 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:12.732331 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:12.732337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:12.732339 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:49:12.732433 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 04:49:12.732493 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260928 04:49:12.732779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 10184
I20260928 04:49:12.732867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:49:12.733068 392 external_mini_cluster.cc:1428] Setting key 652ac6937e9aa39589b8d058e9c8f521
I20260928 04:49:12.733319 10312 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45387
I20260928 04:49:12.733419 10312 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:12.733551 10312 heartbeater.cc:511] Master 127.0.98.62:45387 requested a full tablet report, sending...
I20260928 04:49:12.734073 10124 ts_manager.cc:194] Registered new tserver with Master: cefe5aafc5b14c0bbb912ac7e4fe5153 (127.0.98.1:38535)
I20260928 04:49:12.734093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45387
--builtin_ntp_servers=127.0.98.20:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 04:49:12.816902 10315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:12.817114 10315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:12.817137 10315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:12.818454 10315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:12.818493 10315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:12.818521 10315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:49:12.819845 10315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:12.820027 10315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:12.820266 10315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:12.825421 10320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:12.825457 10315 server_base.cc:1034] running on GCE node
W20260928 04:49:12.825412 10323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.825412 10321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:12.825979 10315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:12.826233 10315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:12.827390 10315 hybrid_clock.cc:648] HybridClock initialized: now 1790570952827379 us; error 28 us; skew 500 ppm
I20260928 04:49:12.828796 10315 webserver.cc:460] Webserver started at http://127.0.98.2:33901/ using document root <none> and password file <none>
I20260928 04:49:12.829051 10315 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:12.829119 10315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:12.829293 10315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:12.830446 10315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "46c471b127d04c17aae749b0d460f8e8"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "461032a24c87c1339cdb2dfa4696536f"
server_key_iv: "b70e857349d7a86061429a4b2c064fb5"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.830847 10315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "46c471b127d04c17aae749b0d460f8e8"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "461032a24c87c1339cdb2dfa4696536f"
server_key_iv: "b70e857349d7a86061429a4b2c064fb5"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.832105 10315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:49:12.832830 10329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.833001 10315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:49:12.833093 10315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "46c471b127d04c17aae749b0d460f8e8"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "461032a24c87c1339cdb2dfa4696536f"
server_key_iv: "b70e857349d7a86061429a4b2c064fb5"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.833173 10315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:12.836565 10315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:12.836843 10315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:12.836982 10315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:12.837241 10315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:12.837556 10315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:12.837610 10315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.837651 10315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:12.837677 10315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.843439 10315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38475
I20260928 04:49:12.843492 10442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38475 every 8 connection(s)
I20260928 04:49:12.843815 10315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:49:12.846858 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.844587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37621 (local address 127.0.98.62:45387)
0928 04:49:12.844631 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:12.844633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:12.844870 (+ 237us) server_negotiation.cc:407] Connection header received
0928 04:49:12.845400 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:12.845401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:12.845408 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:12.845462 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:12.846645 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:12.846646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:12.846646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:49:12.846650 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:12.846663 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:49:12.846668 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:12.846725 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:12.846792 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 04:49:12.846804 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:49:12.847067 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.844570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45387 (local address 127.0.98.2:37621)
0928 04:49:12.844741 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:12.844782 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:12.845310 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:12.845516 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:12.845519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:12.845527 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:12.846497 (+ 970us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:49:12.846498 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:12.846527 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:49:12.846556 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:12.846725 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:12.846727 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:12.846730 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:49:12.846788 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 04:49:12.846835 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260928 04:49:12.847692 10443 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45387
I20260928 04:49:12.847777 10443 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:12.847888 10443 heartbeater.cc:511] Master 127.0.98.62:45387 requested a full tablet report, sending...
I20260928 04:49:12.848242 10124 ts_manager.cc:194] Registered new tserver with Master: 46c471b127d04c17aae749b0d460f8e8 (127.0.98.2:38475)
I20260928 04:49:12.848874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 10315
I20260928 04:49:12.848984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:49:12.849233 392 external_mini_cluster.cc:1428] Setting key 6c3a188866adeb19b6f107d06cbc7945
I20260928 04:49:12.850082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45387
--builtin_ntp_servers=127.0.98.20:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 04:49:12.931730 10446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:12.931957 10446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:12.931978 10446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:12.933224 10446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:12.933279 10446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:12.933307 10446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:49:12.934749 10446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:12.934934 10446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:12.935237 10446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:12.940464 10451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.940542 10452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:12.940510 10454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:12.940702 10446 server_base.cc:1034] running on GCE node
I20260928 04:49:12.940922 10446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:12.941136 10446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:12.942334 10446 hybrid_clock.cc:648] HybridClock initialized: now 1790570952942317 us; error 29 us; skew 500 ppm
I20260928 04:49:12.943715 10446 webserver.cc:460] Webserver started at http://127.0.98.3:34927/ using document root <none> and password file <none>
I20260928 04:49:12.943972 10446 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:12.944023 10446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:12.944191 10446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:12.945400 10446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "af04e2e6ea0947808e28a19132c45180"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "64b8cfdfcfe2306b6ad2f008524a3b73"
server_key_iv: "9babdc9b37bb4a53da02a8db048e3f0e"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.945771 10446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "af04e2e6ea0947808e28a19132c45180"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "64b8cfdfcfe2306b6ad2f008524a3b73"
server_key_iv: "9babdc9b37bb4a53da02a8db048e3f0e"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.947052 10446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:49:12.947902 10460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.948078 10446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:49:12.948182 10446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "af04e2e6ea0947808e28a19132c45180"
format_stamp: "Formatted at 2026-09-28 04:49:12 on dist-test-slave-jccs"
server_key: "64b8cfdfcfe2306b6ad2f008524a3b73"
server_key_iv: "9babdc9b37bb4a53da02a8db048e3f0e"
server_key_version: "encryptionkey@0"
I20260928 04:49:12.948264 10446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:12.951637 10446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:12.951911 10446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:12.952056 10446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:12.952272 10446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:12.952577 10446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:12.952610 10446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.952631 10446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:12.952679 10446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:12.958516 10446 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46173
I20260928 04:49:12.958595 10573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46173 every 8 connection(s)
I20260928 04:49:12.958945 10446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:49:12.962028 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.959725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44923 (local address 127.0.98.62:45387)
0928 04:49:12.959815 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:12.959816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:12.960065 (+ 249us) server_negotiation.cc:407] Connection header received
0928 04:49:12.960520 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:12.960521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:12.960527 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:12.960587 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:12.961826 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:12.961828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:12.961828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:49:12.961833 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:12.961847 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:49:12.961852 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:12.961902 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:12.961961 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 04:49:12.961974 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 04:49:12.962240 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:12.959747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45387 (local address 127.0.98.3:44923)
0928 04:49:12.959938 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:12.959972 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:12.960426 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:12.960656 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:12.960659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:12.960668 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:12.961642 (+ 974us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:49:12.961643 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:12.961671 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:49:12.961699 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:12.961901 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:12.961903 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:12.961906 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:49:12.961957 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:49:12.962007 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260928 04:49:12.962785 10574 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45387
I20260928 04:49:12.962865 10574 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:12.962972 10574 heartbeater.cc:511] Master 127.0.98.62:45387 requested a full tablet report, sending...
I20260928 04:49:12.963297 10124 ts_manager.cc:194] Registered new tserver with Master: af04e2e6ea0947808e28a19132c45180 (127.0.98.3:46173)
I20260928 04:49:12.964973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 10446
I20260928 04:49:12.965049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:49:12.965296 392 external_mini_cluster.cc:1428] Setting key 4e92e5f5e5c81a4140f8da2278601159
I20260928 04:49:12.966051 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 04:49:12.967880 10163 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:12.967425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.0.98.62:45387)
0928 04:49:12.967491 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:12.967493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:12.967588 (+ 95us) server_negotiation.cc:407] Connection header received
0928 04:49:12.967640 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:12.967640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:12.967644 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:12.967696 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:12.967821 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41898: BlockingRecv error: recv got EOF from 127.0.0.1:41898 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:49:12.967970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 10184
I20260928 04:49:12.972911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 10315
I20260928 04:49:12.977502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 10446
I20260928 04:49:12.981877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 10093
2026-09-28T04:49:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (552 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:49:13 dist-test-slave-jccs krb5kdc[10584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:49:13 dist-test-slave-jccs krb5kdc[10584](info): set up 2 sockets
Sep 28 04:49:13 dist-test-slave-jccs krb5kdc[10584](info): commencing operation
krb5kdc: starting...
W20260928 04:49:14.034164 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[10584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:49:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:49:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:49:14.150135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35769
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:49:14.230887 10597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:14.231166 10597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:14.231206 10597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:14.232375 10597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:49:14.232457 10597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:14.232481 10597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:14.232502 10597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:49:14.232529 10597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:49:14.234107 10597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:14.234388 10597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:14.234619 10597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:14.239586 10603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:14.239622 10597 server_base.cc:1034] running on GCE node
W20260928 04:49:14.239722 10605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:14.239599 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:14.240087 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:14.240384 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:14.241549 10597 hybrid_clock.cc:648] HybridClock initialized: now 1790570954241535 us; error 48 us; skew 500 ppm
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[10584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:49:14.244168 10597 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:49:14.244557 10597 webserver.cc:460] Webserver started at http://127.0.98.62:45455/ using document root <none> and password file <none>
I20260928 04:49:14.244741 10597 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:14.244781 10597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:14.244868 10597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:14.245972 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "de91d21f39774ad580be58d8c1cade32"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "b8084ecf48a75661f39b410fe1f73a1e"
server_key_iv: "a47dc27a695991be448fae005f0ff379"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.246284 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "de91d21f39774ad580be58d8c1cade32"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "b8084ecf48a75661f39b410fe1f73a1e"
server_key_iv: "a47dc27a695991be448fae005f0ff379"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.247511 10597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:49:14.248200 10612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.248368 10597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:49:14.248451 10597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "de91d21f39774ad580be58d8c1cade32"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "b8084ecf48a75661f39b410fe1f73a1e"
server_key_iv: "a47dc27a695991be448fae005f0ff379"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.248505 10597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:14.255119 10597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:14.256906 10597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:14.257028 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:14.260742 10597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35769
I20260928 04:49:14.260799 10664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35769 every 8 connection(s)
I20260928 04:49:14.261144 10597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:49:14.262238 10665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:49:14.264431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 10597
I20260928 04:49:14.264534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:49:14.264621 10665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32: Bootstrap starting.
I20260928 04:49:14.264743 392 external_mini_cluster.cc:1428] Setting key 922264e5628d7c4bd9b16b25cbdd1034
I20260928 04:49:14.265451 10665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32: Neither blocks nor log segments found. Creating new log.
I20260928 04:49:14.265808 10665 log.cc:824] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32: Log is configured to *not* fsync() on all Append() calls
I20260928 04:49:14.266471 10665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32: No bootstrap required, opened a new log
I20260928 04:49:14.268256 10665 raft_consensus.cc:348] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de91d21f39774ad580be58d8c1cade32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35769 } }
I20260928 04:49:14.268426 10665 raft_consensus.cc:374] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:49:14.268493 10665 raft_consensus.cc:729] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de91d21f39774ad580be58d8c1cade32, State: Initialized, Role: FOLLOWER
I20260928 04:49:14.268647 10665 consensus_queue.cc:260] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de91d21f39774ad580be58d8c1cade32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35769 } }
I20260928 04:49:14.268746 10665 raft_consensus.cc:388] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:49:14.268791 10665 raft_consensus.cc:482] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:49:14.268839 10665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:49:14.269524 10665 raft_consensus.cc:504] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de91d21f39774ad580be58d8c1cade32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35769 } }
I20260928 04:49:14.269644 10665 leader_election.cc:302] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de91d21f39774ad580be58d8c1cade32; no voters:
I20260928 04:49:14.269809 10665 leader_election.cc:288] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:49:14.269843 10670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:49:14.269974 10670 raft_consensus.cc:686] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [term 1 LEADER]: Becoming Leader. State: Replica: de91d21f39774ad580be58d8c1cade32, State: Running, Role: LEADER
I20260928 04:49:14.270080 10665 sys_catalog.cc:564] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:49:14.270130 10670 consensus_queue.cc:237] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de91d21f39774ad580be58d8c1cade32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35769 } }
I20260928 04:49:14.270514 10671 sys_catalog.cc:455] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de91d21f39774ad580be58d8c1cade32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de91d21f39774ad580be58d8c1cade32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35769 } } }
I20260928 04:49:14.270593 10671 sys_catalog.cc:458] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [sys.catalog]: This master's current role is: LEADER
I20260928 04:49:14.270737 10671 sys_catalog.cc:455] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de91d21f39774ad580be58d8c1cade32. Latest consensus state: current_term: 1 leader_uuid: "de91d21f39774ad580be58d8c1cade32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de91d21f39774ad580be58d8c1cade32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35769 } } }
I20260928 04:49:14.270785 10671 sys_catalog.cc:458] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32 [sys.catalog]: This master's current role is: LEADER
I20260928 04:49:14.271341 10677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[10584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:49:14.271936 10677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:49:14.273512 10677 catalog_manager.cc:1347] Generated new cluster ID: e545f1cf24cb4ad3b9b58d7f5ee13d90
I20260928 04:49:14.273561 10677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:49:14.275760 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.265382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.0.98.62:35769)
0928 04:49:14.265554 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.265564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.265594 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:49:14.266084 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.266091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.266141 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.266298 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:14.266980 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.267858 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.268671 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.269272 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.269399 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.272355 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:14.272359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:14.272359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:49:14.272372 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:14.274214 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:14.274569 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:14.274572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:14.274572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:14.274666 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:14.274877 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:14.274878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:14.274878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:14.275113 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:14.275289 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.275364 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.275526 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260928 04:49:14.283761 10677 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:49:14.284489 10677 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:49:14.289372 10677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32: Generated new TSK 0
I20260928 04:49:14.289582 10677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:49:14.315838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35769
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:49:14.398121 10691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:14.398367 10691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:14.398413 10691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:14.399725 10691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:14.399799 10691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:14.399847 10691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:49:14.401485 10691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:14.401755 10691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:14.401993 10691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:14.407699 10696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:14.407708 10699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:14.407706 10697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:14.408059 10691 server_base.cc:1034] running on GCE node
I20260928 04:49:14.408269 10691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:14.408536 10691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:14.409698 10691 hybrid_clock.cc:648] HybridClock initialized: now 1790570954409684 us; error 32 us; skew 500 ppm
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[10584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:49:14.412300 10691 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:49:14.412797 10691 webserver.cc:460] Webserver started at http://127.0.98.1:33215/ using document root <none> and password file <none>
I20260928 04:49:14.413050 10691 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:14.413278 10691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:14.413460 10691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:14.414557 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "605e29ffeb7955afeed3723740f33300"
server_key_iv: "d50152e7e4825b06e076464f1e4054e0"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.414938 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "605e29ffeb7955afeed3723740f33300"
server_key_iv: "d50152e7e4825b06e076464f1e4054e0"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.416328 10691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:49:14.417100 10706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.417311 10691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:49:14.417438 10691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "605e29ffeb7955afeed3723740f33300"
server_key_iv: "d50152e7e4825b06e076464f1e4054e0"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.417527 10691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:14.425345 10691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:14.427507 10691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:14.427686 10691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:14.427919 10691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:14.428238 10691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:14.428296 10691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.428336 10691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:14.428361 10691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.434820 10691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33259
I20260928 04:49:14.434863 10819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33259 every 8 connection(s)
I20260928 04:49:14.435189 10691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:49:14.441233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 10691
I20260928 04:49:14.441326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[10584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:49:14.441532 392 external_mini_cluster.cc:1428] Setting key 4a7403d5c1537f85c4f9581d6ad9192a
I20260928 04:49:14.445348 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.435891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47853 (local address 127.0.98.62:35769)
0928 04:49:14.435943 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.435946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.436169 (+ 223us) server_negotiation.cc:407] Connection header received
0928 04:49:14.436621 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.436622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.436632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.436684 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:14.437674 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.438168 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.439148 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.439342 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.439391 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.441879 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:14.441881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:14.441881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:49:14.441889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:14.443696 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:14.444257 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:14.444258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:14.444259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:14.444312 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:14.444596 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:14.444597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:14.444597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:14.444701 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:14.444752 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.445127 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.445272 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:49:14.445560 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.435878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35769 (local address 127.0.98.1:47853)
0928 04:49:14.436037 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:14.436073 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:14.436538 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:14.436738 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:14.436741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:14.437383 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:14.437581 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.437585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.438249 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.438249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.439074 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.439076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.439415 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.439416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.439520 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.440074 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:14.440085 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:14.441785 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:14.443789 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:14.443797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:14.443800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:14.444177 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:14.444384 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:14.444385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:14.444385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:14.444529 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:14.444767 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:14.444773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:14.444949 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 04:49:14.445072 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 04:49:14.445277 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260928 04:49:14.446107 10820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35769
I20260928 04:49:14.446218 10820 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:14.446413 10820 heartbeater.cc:511] Master 127.0.98.62:35769 requested a full tablet report, sending...
I20260928 04:49:14.446976 10629 ts_manager.cc:194] Registered new tserver with Master: 74e1e2315ffe4afcb6c30d22ab4d1f9b (127.0.98.1:33259)
I20260928 04:49:14.447602 10629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47853
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:49:14.468603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35769
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:49:14.552337 10825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:14.552587 10825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:14.552626 10825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:14.553963 10825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:14.554037 10825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:14.554085 10825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:49:14.555603 10825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:14.555887 10825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:14.556136 10825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:14.561232 10830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:14.561252 10831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:14.561340 10825 server_base.cc:1034] running on GCE node
W20260928 04:49:14.561244 10833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:14.561648 10825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:14.561852 10825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:14.562994 10825 hybrid_clock.cc:648] HybridClock initialized: now 1790570954562981 us; error 29 us; skew 500 ppm
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[10584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:49:14.565620 10825 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:49:14.566145 10825 webserver.cc:460] Webserver started at http://127.0.98.2:42853/ using document root <none> and password file <none>
I20260928 04:49:14.566393 10825 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:14.566442 10825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:14.566599 10825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:14.567844 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "814fba12fda44930b93ff539ebe99bbc"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "9be7ae7538642555cf0a2b4badd1041a"
server_key_iv: "5ba70b26c52708d4b839f1572ce51a0c"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.568274 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "814fba12fda44930b93ff539ebe99bbc"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "9be7ae7538642555cf0a2b4badd1041a"
server_key_iv: "5ba70b26c52708d4b839f1572ce51a0c"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.569792 10825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:49:14.570655 10840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.570859 10825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:49:14.570968 10825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "814fba12fda44930b93ff539ebe99bbc"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "9be7ae7538642555cf0a2b4badd1041a"
server_key_iv: "5ba70b26c52708d4b839f1572ce51a0c"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.571044 10825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:14.617940 10825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:14.620355 10825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:14.620572 10825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:14.620838 10825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:14.621268 10825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:14.621327 10825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.621372 10825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:14.621398 10825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.627791 10825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46449
I20260928 04:49:14.627844 10953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46449 every 8 connection(s)
I20260928 04:49:14.628217 10825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:49:14.634552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 10825
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[10584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:49:14.634686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:49:14.634882 392 external_mini_cluster.cc:1428] Setting key b1cd845f124e0f7fe520016187fb2e30
I20260928 04:49:14.638190 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.629242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52541 (local address 127.0.98.62:35769)
0928 04:49:14.629309 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.629312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.629508 (+ 196us) server_negotiation.cc:407] Connection header received
0928 04:49:14.629964 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.629964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.629973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.630033 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:14.630972 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.631471 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.632555 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.632733 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.632786 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.635185 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:14.635186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:14.635187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:49:14.635195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:14.636701 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:14.637149 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:14.637150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:14.637150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:14.637242 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:14.637519 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:14.637520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:14.637521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:14.637639 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:14.637712 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.638004 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.638092 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:49:14.638404 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.629136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35769 (local address 127.0.98.2:52541)
0928 04:49:14.629364 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:14.629407 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:14.629881 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:14.630095 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:14.630099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:14.630673 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:14.630883 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.630889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.631594 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.631595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.632445 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.632448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.632807 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.632808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.632908 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.633476 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:14.633484 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:14.635080 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:14.636790 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:14.636798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:14.636801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:14.637068 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:14.637323 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:14.637323 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:14.637324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:14.637444 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:14.637717 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:14.637723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:14.637870 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 04:49:14.637981 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 04:49:14.638158 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":156,"threads_started":1}
I20260928 04:49:14.639037 10954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35769
I20260928 04:49:14.639207 10954 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:14.639410 10954 heartbeater.cc:511] Master 127.0.98.62:35769 requested a full tablet report, sending...
I20260928 04:49:14.639868 10629 ts_manager.cc:194] Registered new tserver with Master: 814fba12fda44930b93ff539ebe99bbc (127.0.98.2:46449)
I20260928 04:49:14.640413 10629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52541
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:49:14.660583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35769
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:49:14.745747 10959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:14.746007 10959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:14.746049 10959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:14.747226 10959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:14.747306 10959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:14.747354 10959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:49:14.748839 10959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:14.749135 10959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:14.749394 10959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:14.754317 10965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:14.754354 10964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:14.754500 10959 server_base.cc:1034] running on GCE node
W20260928 04:49:14.754408 10967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:14.754714 10959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:14.754967 10959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:14.756168 10959 hybrid_clock.cc:648] HybridClock initialized: now 1790570954756153 us; error 30 us; skew 500 ppm
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[10584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:49:14.758847 10959 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:49:14.759606 10959 webserver.cc:460] Webserver started at http://127.0.98.3:45409/ using document root <none> and password file <none>
I20260928 04:49:14.759900 10959 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:14.759999 10959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:14.760152 10959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:49:14.761310 10959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "370a4540c62046c1b4564caf404c0593"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "679a328ee515bbd71e99d3a41ca8ebd0"
server_key_iv: "5980ab893ea05a5210d2c250d7e4cc1d"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.761684 10959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "370a4540c62046c1b4564caf404c0593"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "679a328ee515bbd71e99d3a41ca8ebd0"
server_key_iv: "5980ab893ea05a5210d2c250d7e4cc1d"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.762971 10959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 04:49:14.763684 10974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.763856 10959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.763957 10959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "370a4540c62046c1b4564caf404c0593"
format_stamp: "Formatted at 2026-09-28 04:49:14 on dist-test-slave-jccs"
server_key: "679a328ee515bbd71e99d3a41ca8ebd0"
server_key_iv: "5980ab893ea05a5210d2c250d7e4cc1d"
server_key_version: "encryptionkey@0"
I20260928 04:49:14.764045 10959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:14.786736 10959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:14.788805 10959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:14.789004 10959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:14.789317 10959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:14.789703 10959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:14.789762 10959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.789821 10959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:14.789850 10959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:14.795980 10959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42619
I20260928 04:49:14.796052 11087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42619 every 8 connection(s)
I20260928 04:49:14.796384 10959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[10584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:49:14.805660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 10959
I20260928 04:49:14.805763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:49:14.805925 392 external_mini_cluster.cc:1428] Setting key 4db018a4cf3f91fd34b3f98e3682c1fa
I20260928 04:49:14.805964 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.797134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37327 (local address 127.0.98.62:35769)
0928 04:49:14.797249 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.797252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.797419 (+ 167us) server_negotiation.cc:407] Connection header received
0928 04:49:14.797937 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.797938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.797950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.798009 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:14.798964 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.799468 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.800493 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.800689 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.800756 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.802963 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:14.802965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:14.802965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:49:14.802971 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:14.804374 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:14.804801 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:14.804802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:14.804802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:14.804879 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:14.805317 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:14.805318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:14.805319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:14.805432 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:14.805491 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.805785 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.805875 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 04:49:14.806247 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.797115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35769 (local address 127.0.98.3:37327)
0928 04:49:14.797290 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:14.797334 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:14.797833 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:14.798088 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:14.798091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:14.798671 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:14.798859 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.798863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.799567 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.799568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.800375 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.800377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.800770 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.800771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.800869 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.801497 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:14.801505 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:14.802876 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:14.804460 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:14.804466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:14.804468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:14.804709 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:14.805039 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:14.805040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:14.805041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:14.805230 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:14.805505 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:14.805510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:14.805634 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 04:49:14.805745 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 04:49:14.805905 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260928 04:49:14.806774 11088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35769
I20260928 04:49:14.806875 11088 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:14.807148 11088 heartbeater.cc:511] Master 127.0.98.62:35769 requested a full tablet report, sending...
I20260928 04:49:14.807524 10629 ts_manager.cc:194] Registered new tserver with Master: 370a4540c62046c1b4564caf404c0593 (127.0.98.3:42619)
I20260928 04:49:14.807855 10629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37327
I20260928 04:49:14.817937 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:49:14.825317 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.819291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.0.98.62:35769)
0928 04:49:14.819349 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.819350 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.819440 (+ 90us) server_negotiation.cc:407] Connection header received
0928 04:49:14.819478 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.819479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.819487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.819531 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:14.820095 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.820632 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.821440 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.821624 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.821680 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.822913 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:14.822914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:14.822914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:49:14.822920 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:14.824218 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:14.824558 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:14.824559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:14.824559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:14.824606 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:14.824768 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:14.824769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:14.824769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:14.824905 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:14.824976 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.825081 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.825137 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:49:14.827035 10629 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:49:14.828547 10629 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:49:14.835280 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.831117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42619 (local address 127.0.0.1:49370)
0928 04:49:14.831433 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:14.831442 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:14.831491 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:14.831793 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:14.831794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:14.831797 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:49:14.831849 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.831851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.832711 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.832713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.833949 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.833952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.834946 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.834948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.835026 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.835042 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 04:49:14.835113 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:49:14.835193 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":79,"threads_started":1}
I20260928 04:49:14.835309 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.831055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.0.98.1:33259)
0928 04:49:14.831275 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.831280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.831448 (+ 168us) server_negotiation.cc:407] Connection header received
0928 04:49:14.831682 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.831686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.831702 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.831775 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:49:14.831961 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.833066 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.834257 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.834895 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.834999 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.835056 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.835167 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.835213 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":110,"threads_started":1}
I20260928 04:49:14.835371 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.831279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.0.98.2:46449)
0928 04:49:14.831455 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.831459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.831468 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:49:14.831518 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.831520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.831538 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.831622 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:49:14.831900 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.833039 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.834313 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.835034 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.835126 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.835179 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.835243 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.835279 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260928 04:49:14.835404 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.830932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46449 (local address 127.0.0.1:58594)
0928 04:49:14.831075 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:14.831121 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:14.831193 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:14.831738 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:14.831741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:14.831743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:49:14.831815 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.831817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.833158 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.833159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.834242 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.834245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.835123 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.835124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.835178 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.835182 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:49:14.835232 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 04:49:14.835279 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":53,"threads_started":1}
I20260928 04:49:14.835568 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.831213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.0.98.3:42619)
0928 04:49:14.831386 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.831390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.831485 (+ 95us) server_negotiation.cc:407] Connection header received
0928 04:49:14.831641 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.831643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.831659 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.831742 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:49:14.831906 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.832604 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.834040 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.834785 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.834854 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.834912 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.835443 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.835495 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260928 04:49:14.836052 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.830932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33259 (local address 127.0.0.1:38568)
0928 04:49:14.831378 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:14.831389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:14.831457 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:14.831848 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:14.831848 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:14.831851 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:49:14.831886 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.831887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.833206 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.833207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.834161 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.834163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.835001 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.835002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.835066 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.835070 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:49:14.835124 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:49:14.835982 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":809,"spinlock_wait_cycles":896,"thread_start_us":80,"threads_started":1}
I20260928 04:49:14.836438 11022 tablet_service.cc:1467] Processing CreateTablet for tablet 0b72c158e2d14d708f8d4eff3814cf8d (DEFAULT_TABLE table=test-table [id=070f933a0ba9476d9d607f46f59b9191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:49:14.836683 10888 tablet_service.cc:1467] Processing CreateTablet for tablet 0b72c158e2d14d708f8d4eff3814cf8d (DEFAULT_TABLE table=test-table [id=070f933a0ba9476d9d607f46f59b9191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:49:14.836802 11022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b72c158e2d14d708f8d4eff3814cf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:49:14.836985 10888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b72c158e2d14d708f8d4eff3814cf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:49:14.837823 10754 tablet_service.cc:1467] Processing CreateTablet for tablet 0b72c158e2d14d708f8d4eff3814cf8d (DEFAULT_TABLE table=test-table [id=070f933a0ba9476d9d607f46f59b9191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:49:14.838157 10754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b72c158e2d14d708f8d4eff3814cf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:49:14.840255 11105 tablet_bootstrap.cc:492] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: Bootstrap starting.
I20260928 04:49:14.840507 11106 tablet_bootstrap.cc:492] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: Bootstrap starting.
I20260928 04:49:14.840868 11107 tablet_bootstrap.cc:492] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b: Bootstrap starting.
I20260928 04:49:14.841264 11106 tablet_bootstrap.cc:654] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: Neither blocks nor log segments found. Creating new log.
I20260928 04:49:14.841543 11107 tablet_bootstrap.cc:654] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b: Neither blocks nor log segments found. Creating new log.
I20260928 04:49:14.841828 11105 tablet_bootstrap.cc:654] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: Neither blocks nor log segments found. Creating new log.
I20260928 04:49:14.841866 11106 log.cc:824] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: Log is configured to *not* fsync() on all Append() calls
I20260928 04:49:14.842152 11107 log.cc:824] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b: Log is configured to *not* fsync() on all Append() calls
I20260928 04:49:14.842432 11105 log.cc:824] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: Log is configured to *not* fsync() on all Append() calls
I20260928 04:49:14.842707 11106 tablet_bootstrap.cc:492] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: No bootstrap required, opened a new log
I20260928 04:49:14.842765 11106 ts_tablet_manager.cc:1397] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:49:14.842939 11107 tablet_bootstrap.cc:492] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b: No bootstrap required, opened a new log
I20260928 04:49:14.843025 11107 ts_tablet_manager.cc:1397] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:49:14.844208 11106 raft_consensus.cc:348] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 } } peers { permanent_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33259 } } peers { permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 } }
I20260928 04:49:14.844352 11106 raft_consensus.cc:374] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:49:14.844374 11106 raft_consensus.cc:729] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814fba12fda44930b93ff539ebe99bbc, State: Initialized, Role: FOLLOWER
I20260928 04:49:14.844331 11107 raft_consensus.cc:348] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 } } peers { permanent_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33259 } } peers { permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 } }
I20260928 04:49:14.844558 11107 raft_consensus.cc:374] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:49:14.844566 11106 consensus_queue.cc:260] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 } } peers { permanent_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33259 } } peers { permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 } }
I20260928 04:49:14.844623 11107 raft_consensus.cc:729] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e1e2315ffe4afcb6c30d22ab4d1f9b, State: Initialized, Role: FOLLOWER
I20260928 04:49:14.844743 11107 consensus_queue.cc:260] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 } } peers { permanent_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33259 } } peers { permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 } }
I20260928 04:49:14.844844 11106 ts_tablet_manager.cc:1428] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:49:14.844974 10954 heartbeater.cc:503] Master 127.0.98.62:35769 was elected leader, sending a full tablet report...
I20260928 04:49:14.845067 11107 ts_tablet_manager.cc:1428] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:49:14.845315 10820 heartbeater.cc:503] Master 127.0.98.62:35769 was elected leader, sending a full tablet report...
I20260928 04:49:14.845804 11105 tablet_bootstrap.cc:492] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: No bootstrap required, opened a new log
I20260928 04:49:14.845875 11105 ts_tablet_manager.cc:1397] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 04:49:14.847806 11105 raft_consensus.cc:348] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 } } peers { permanent_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33259 } } peers { permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 } }
I20260928 04:49:14.848014 11105 raft_consensus.cc:374] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:49:14.848065 11105 raft_consensus.cc:729] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370a4540c62046c1b4564caf404c0593, State: Initialized, Role: FOLLOWER
I20260928 04:49:14.848196 11105 consensus_queue.cc:260] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 } } peers { permanent_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33259 } } peers { permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 } }
I20260928 04:49:14.848486 11105 ts_tablet_manager.cc:1428] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:49:14.848521 11088 heartbeater.cc:503] Master 127.0.98.62:35769 was elected leader, sending a full tablet report...
W20260928 04:49:14.878816 10955 tablet.cc:2369] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:49:14.883083 11112 raft_consensus.cc:482] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:49:14.883203 11112 raft_consensus.cc:504] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 } } peers { permanent_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33259 } } peers { permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 } }
I20260928 04:49:14.883563 11112 leader_election.cc:288] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 814fba12fda44930b93ff539ebe99bbc (127.0.98.2:46449), 370a4540c62046c1b4564caf404c0593 (127.0.98.3:42619)
I20260928 04:49:14.886370 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.883829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47051 (local address 127.0.98.2:46449)
0928 04:49:14.883878 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.883880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.883892 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:49:14.883926 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.883927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.883940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.884026 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:49:14.884201 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.884675 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.885622 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.885988 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.886092 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.886103 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.886254 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.886294 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:49:14.886374 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.883754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46449 (local address 127.0.98.1:47051)
0928 04:49:14.883814 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:14.883826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:14.883871 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:14.884033 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:14.884034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:14.884035 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:49:14.884121 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.884124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.884763 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.884764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.885516 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.885518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.886084 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.886085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.886172 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.886175 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:49:14.886246 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:49:14.886298 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 04:49:14.886574 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.883857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46867 (local address 127.0.98.3:42619)
0928 04:49:14.883914 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.883916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.884045 (+ 129us) server_negotiation.cc:407] Connection header received
0928 04:49:14.884106 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.884107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.884121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.884197 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:49:14.884365 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.884807 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.885818 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.886190 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.886278 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.886288 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.886450 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.886503 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:49:14.886662 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.883780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42619 (local address 127.0.98.1:46867)
0928 04:49:14.883972 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:14.883980 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:14.884055 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:14.884202 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:14.884202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:14.884204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:49:14.884274 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.884275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.884901 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.884901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.885727 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:14.885729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.886281 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.886282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:14.886356 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.886362 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:49:14.886433 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:49:14.886568 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":124,"threads_started":1}
I20260928 04:49:14.886875 10908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b72c158e2d14d708f8d4eff3814cf8d" candidate_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814fba12fda44930b93ff539ebe99bbc" is_pre_election: true
I20260928 04:49:14.886976 11042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b72c158e2d14d708f8d4eff3814cf8d" candidate_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370a4540c62046c1b4564caf404c0593" is_pre_election: true
I20260928 04:49:14.887048 10908 raft_consensus.cc:2454] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e1e2315ffe4afcb6c30d22ab4d1f9b in term 0.
I20260928 04:49:14.887089 11042 raft_consensus.cc:2454] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74e1e2315ffe4afcb6c30d22ab4d1f9b in term 0.
I20260928 04:49:14.887306 10710 leader_election.cc:302] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370a4540c62046c1b4564caf404c0593, 74e1e2315ffe4afcb6c30d22ab4d1f9b; no voters:
I20260928 04:49:14.887480 11112 raft_consensus.cc:2781] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:49:14.887545 11112 raft_consensus.cc:482] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:49:14.887562 11112 raft_consensus.cc:3037] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:49:14.888298 11112 raft_consensus.cc:504] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 } } peers { permanent_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33259 } } peers { permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 } }
I20260928 04:49:14.888450 11112 leader_election.cc:288] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [CANDIDATE]: Term 1 election: Requested vote from peers 814fba12fda44930b93ff539ebe99bbc (127.0.98.2:46449), 370a4540c62046c1b4564caf404c0593 (127.0.98.3:42619)
I20260928 04:49:14.888624 11042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b72c158e2d14d708f8d4eff3814cf8d" candidate_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370a4540c62046c1b4564caf404c0593"
I20260928 04:49:14.888624 10908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b72c158e2d14d708f8d4eff3814cf8d" candidate_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "814fba12fda44930b93ff539ebe99bbc"
I20260928 04:49:14.888702 10908 raft_consensus.cc:3037] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:49:14.888702 11042 raft_consensus.cc:3037] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:49:14.889640 10908 raft_consensus.cc:2454] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e1e2315ffe4afcb6c30d22ab4d1f9b in term 1.
I20260928 04:49:14.889657 11042 raft_consensus.cc:2454] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74e1e2315ffe4afcb6c30d22ab4d1f9b in term 1.
I20260928 04:49:14.889817 10709 leader_election.cc:302] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e1e2315ffe4afcb6c30d22ab4d1f9b, 814fba12fda44930b93ff539ebe99bbc; no voters:
I20260928 04:49:14.889938 11112 raft_consensus.cc:2781] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:49:14.890108 11112 raft_consensus.cc:686] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 1 LEADER]: Becoming Leader. State: Replica: 74e1e2315ffe4afcb6c30d22ab4d1f9b, State: Running, Role: LEADER
I20260928 04:49:14.890221 11112 consensus_queue.cc:237] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 } } peers { permanent_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33259 } } peers { permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 } }
I20260928 04:49:14.891093 10629 catalog_manager.cc:5495] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 74e1e2315ffe4afcb6c30d22ab4d1f9b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74e1e2315ffe4afcb6c30d22ab4d1f9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 } health_report { overall_health: UNKNOWN } } }
I20260928 04:49:14.896615 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.894442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.0.98.1:33259)
0928 04:49:14.894511 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:14.894514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:14.894536 (+ 22us) server_negotiation.cc:407] Connection header received
0928 04:49:14.894571 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:14.894571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:14.894586 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:14.894639 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:49:14.894751 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.895195 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.895916 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:14.896088 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:14.896139 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:14.896281 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:49:14.896377 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:49:14.896436 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:14.896470 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 04:49:14.896485 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 04:49:14.898427 11042 raft_consensus.cc:1261] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593 [term 1 FOLLOWER]: Refusing update from remote peer 74e1e2315ffe4afcb6c30d22ab4d1f9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:49:14.898432 10908 raft_consensus.cc:1261] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc [term 1 FOLLOWER]: Refusing update from remote peer 74e1e2315ffe4afcb6c30d22ab4d1f9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:49:14.898835 11112 consensus_queue.cc:1035] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "370a4540c62046c1b4564caf404c0593" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:49:14.898991 11115 consensus_queue.cc:1035] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "814fba12fda44930b93ff539ebe99bbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:49:14.901353 11118 mvcc.cc:205] Tried to move back new op lower bound from 7334178631260975104 to 7334178631230676992. Current Snapshot: MvccSnapshot[applied={T|T < 7334178631260975104}]
I20260928 04:49:14.901741 11119 mvcc.cc:205] Tried to move back new op lower bound from 7334178631260975104 to 7334178631230676992. Current Snapshot: MvccSnapshot[applied={T|T < 7334178631260975104}]
I20260928 04:49:14.901942 11120 mvcc.cc:205] Tried to move back new op lower bound from 7334178631260975104 to 7334178631230676992. Current Snapshot: MvccSnapshot[applied={T|T < 7334178631260975104}]
I20260928 04:49:14.904606 10629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:49:14.904722 10629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:49:14.905566 10629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de91d21f39774ad580be58d8c1cade32: Sending DeleteTablet for 3 replicas of tablet 0b72c158e2d14d708f8d4eff3814cf8d
I20260928 04:49:14.905902 10754 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b72c158e2d14d708f8d4eff3814cf8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:49:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38568
I20260928 04:49:14.905932 10888 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b72c158e2d14d708f8d4eff3814cf8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:49:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58594
I20260928 04:49:14.906028 11022 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b72c158e2d14d708f8d4eff3814cf8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:49:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49370
I20260928 04:49:14.906271 11127 tablet_replica.cc:330] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b: stopping tablet replica
I20260928 04:49:14.906272 11129 tablet_replica.cc:330] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: stopping tablet replica
I20260928 04:49:14.906384 11129 raft_consensus.cc:2229] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:49:14.906404 11127 raft_consensus.cc:2229] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:49:14.906527 11127 raft_consensus.cc:2258] T 0b72c158e2d14d708f8d4eff3814cf8d P 74e1e2315ffe4afcb6c30d22ab4d1f9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:49:14.906580 11129 raft_consensus.cc:2258] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:49:14.906569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 10691
I20260928 04:49:14.907435 11128 tablet_replica.cc:330] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: stopping tablet replica
I20260928 04:49:14.907572 11128 raft_consensus.cc:2229] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:49:14.907684 11128 raft_consensus.cc:2258] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:49:14.908227 11128 ts_tablet_manager.cc:1905] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:49:14.908740 11129 ts_tablet_manager.cc:1905] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:49:14.910202 11129 ts_tablet_manager.cc:1918] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:49:14.910298 11129 log.cc:1196] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/0b72c158e2d14d708f8d4eff3814cf8d
I20260928 04:49:14.910336 11128 ts_tablet_manager.cc:1918] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:49:14.910449 11128 log.cc:1196] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/0b72c158e2d14d708f8d4eff3814cf8d
I20260928 04:49:14.910633 11129 ts_tablet_manager.cc:1939] T 0b72c158e2d14d708f8d4eff3814cf8d P 370a4540c62046c1b4564caf404c0593: Deleting consensus metadata
I20260928 04:49:14.910748 11128 ts_tablet_manager.cc:1939] T 0b72c158e2d14d708f8d4eff3814cf8d P 814fba12fda44930b93ff539ebe99bbc: Deleting consensus metadata
I20260928 04:49:14.911099 10616 catalog_manager.cc:4841] TS 370a4540c62046c1b4564caf404c0593 (127.0.98.3:42619): tablet 0b72c158e2d14d708f8d4eff3814cf8d (table test-table [id=070f933a0ba9476d9d607f46f59b9191]) successfully deleted
I20260928 04:49:14.911285 10615 catalog_manager.cc:4841] TS 814fba12fda44930b93ff539ebe99bbc (127.0.98.2:46449): tablet 0b72c158e2d14d708f8d4eff3814cf8d (table test-table [id=070f933a0ba9476d9d607f46f59b9191]) successfully deleted
W20260928 04:49:14.912388 10616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33259 (error 108)
I20260928 04:49:14.912670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 10825
I20260928 04:49:14.913354 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:14.912802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33259 (local address 127.0.0.1:38580)
0928 04:49:14.913262 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:14.913312 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
W20260928 04:49:14.913517 10616 catalog_manager.cc:4584] TS 74e1e2315ffe4afcb6c30d22ab4d1f9b (127.0.98.1:33259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b72c158e2d14d708f8d4eff3814cf8d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33259: connect: Connection refused (error 111)
I20260928 04:49:14.918437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 10959
I20260928 04:49:14.922919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 10597
2026-09-28T04:49:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1942 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[11133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[11133](info): set up 2 sockets
Sep 28 04:49:14 dist-test-slave-jccs krb5kdc[11133](info): commencing operation
krb5kdc: starting...
W20260928 04:49:15.974200 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:49:16 dist-test-slave-jccs krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:49:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 04:49:16.138257 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNuqiey/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/postgres -l logfile start
2026-09-28 04:49:18.614 UTC [11163] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 04:49:18.614 UTC [11163] LOG: listening on IPv4 address "127.0.98.20", port 40969
2026-09-28 04:49:18.617 UTC [11163] LOG: listening on Unix socket "/tmp/.s.PGSQL.40969"
2026-09-28 04:49:18.647 UTC [11166] LOG: database system was shut down at 2026-09-28 04:49:17 UTC
2026-09-28 04:49:18.651 UTC [11163] LOG: database system is ready to accept connections
I20260928 04:49:19.582387 392 mini_postgres.cc:96] Postgres bound to 40969
2026-09-28 04:49:19.588 UTC [11174] FATAL: database "slave" does not exist
127.0.98.20:40969 - accepting connections
I20260928 04:49:19.589499 392 mini_ranger.cc:161] Starting Ranger...
I20260928 04:49:19.604399 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 04:49:20.088191 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 04:49:20.088254 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-admin
2026-09-28 04:49:20,352 [I] DB FLAVOR :POSTGRES
2026-09-28 04:49:20,352 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:49:20,352 [I] Checking connection..
2026-09-28 04:49:20,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 04:49:20,597 [I] Checking connection passed.
2026-09-28 04:49:20,598 [I] --------- Verifying version history table ---------
2026-09-28 04:49:20,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:49:20,844 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:49:20,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:49:21,101 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:49:21,102 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 04:49:21,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 04:49:21.348 UTC [11238] WARNING: there is no transaction in progress
2026-09-28 04:49:21,354 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 04:49:21,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:49:21,602 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 04:49:21,602 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:49:21,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 04:49:21,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 04:49:22,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jccs.c.gcp-upstream.internal','N') ;"
2026-09-28 04:49:22,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:49:22,586 [I] Table x_portal_user does not exist in database ranger
2026-09-28 04:49:22,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:49:22,817 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 04:49:22,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:49:23,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:49:23,287 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 04:49:23,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 04:49:24.238 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.253 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.269 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.285 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.300 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.320 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.375 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.382 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.395 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.405 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.415 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.428 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.438 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.444 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.452 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:24.458 UTC [11346] WARNING: there is no transaction in progress
2026-09-28 04:49:25,081 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 04:49:25,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 04:49:25,333 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 04:49:25,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:49:25,580 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 04:49:25,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:49:25,842 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 04:49:25,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:49:26,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:49:26,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jccs.c.gcp-upstream.internal';"
2026-09-28 04:49:26,622 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 04:49:26,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 04:49:26,858 [I] DB_PATCHES have already been applied
I20260928 04:49:26.863677 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNuqiey/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 04:49:26.863741 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 04:49:26.866882 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:42541
Sep 28, 2026 4:49:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:49:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:49:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:49:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:49:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:49:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:49:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 04:49:27.981495 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.392s user 0.000s sys 0.009s
Sep 28, 2026 4:49:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:49:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 4:49:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:49:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:49:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 04:49:49.194852 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 04:49:49 dist-test-slave-jccs krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570989, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 04:49:49.273677 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 04:49:49.282509 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 04:49:50.077746 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 04:49:50.077822 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-kms
I20260928 04:49:50.077842 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:40969
2026-09-28 04:49:50,244 [I] DB FLAVOR :POSTGRES
2026-09-28 04:49:50,244 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:49:50,244 [I] Checking connection
2026-09-28 04:49:50,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 04:49:50,483 [I] connection success
2026-09-28 04:49:50,483 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 04:49:50,483 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 04:49:50,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 04:49:50,720 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 04:49:50,721 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:49:50,721 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 04:49:50,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 04:49:50,960 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 04:49:51.375747 392 mini_ranger_kms.cc:310] Created kms service
I20260928 04:49:51.528957 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 04:49:51.570760 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 04:49:51 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570989, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:49:51.941247 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 04:49:51.941356 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 04:49:51.941377 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 04:49:51.944763 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11526
I20260928 04:49:51.944857 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:41325
Sep 28, 2026 4:49:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:49:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:49:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:49:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:49:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:49:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 04:49:54.089046 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.815s user 0.013s sys 0.137s
I20260928 04:49:54.089159 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 04:49:54.089253 392 mini_ranger_kms.cc:401] 127.0.98.20:41325/kms/v1/keys
Sep 28 04:49:54 dist-test-slave-jccs krb5kdc[11133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790570994, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 04:49:54 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570994, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 04:49:54 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570994, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 04:49:56 dist-test-slave-jccs krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:49:57.016249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36447
--ranger_config_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41325/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:49:57.095983 11577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:57.096153 11577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:57.096174 11577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:57.097313 11577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:49:57.097350 11577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:57.097364 11577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:57.097376 11577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:49:57.097415 11577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:49:57.098918 11577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:41325/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36447
--ranger_config_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:57.099150 11577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:57.099395 11577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:57.104112 11583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:57.104175 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:57.104218 11577 server_base.cc:1034] running on GCE node
W20260928 04:49:57.104310 11582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:57.104552 11577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:57.104826 11577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:57.105999 11577 hybrid_clock.cc:648] HybridClock initialized: now 1790570997105989 us; error 26 us; skew 500 ppm
Sep 28 04:49:57 dist-test-slave-jccs krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:49:57.115106 11577 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:49:57.115593 11577 webserver.cc:460] Webserver started at http://127.0.98.62:38987/ using document root <none> and password file <none>
I20260928 04:49:57.115854 11577 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:57.115888 11577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:57.116001 11577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:49:57 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:49:57.173336 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "434da8180e9c4bf297c05b35166a5384"
format_stamp: "Formatted at 2026-09-28 04:49:57 on dist-test-slave-jccs"
server_key: "c50c5bc9dfb7cac6e96374c364240134"
server_key_iv: "d511ab9a1022ff039bac9383e1bab673"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:57.173734 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "434da8180e9c4bf297c05b35166a5384"
format_stamp: "Formatted at 2026-09-28 04:49:57 on dist-test-slave-jccs"
server_key: "c50c5bc9dfb7cac6e96374c364240134"
server_key_iv: "d511ab9a1022ff039bac9383e1bab673"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:57.175148 11577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:49:57.189241 11592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:57.189409 11577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:49:57.189498 11577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "434da8180e9c4bf297c05b35166a5384"
format_stamp: "Formatted at 2026-09-28 04:49:57 on dist-test-slave-jccs"
server_key: "c50c5bc9dfb7cac6e96374c364240134"
server_key_iv: "d511ab9a1022ff039bac9383e1bab673"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:57.189597 11577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:57.198187 11577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:57.200263 11577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:57.200407 11577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:57.204941 11577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36447
I20260928 04:49:57.204986 11644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36447 every 8 connection(s)
I20260928 04:49:57.205391 11577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:49:57.206025 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:49:57.208153 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384: Bootstrap starting.
I20260928 04:49:57.208719 11645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384: Neither blocks nor log segments found. Creating new log.
I20260928 04:49:57.208998 11645 log.cc:824] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384: Log is configured to *not* fsync() on all Append() calls
I20260928 04:49:57.209664 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384: No bootstrap required, opened a new log
I20260928 04:49:57.211316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 11577
I20260928 04:49:57.211418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:49:57.211616 392 external_mini_cluster.cc:1428] Setting key ef2671e3f59de0ecc3495ee94e0e2b1e
I20260928 04:49:57.211493 11645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434da8180e9c4bf297c05b35166a5384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260928 04:49:57.211668 11645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:49:57.211738 11645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 434da8180e9c4bf297c05b35166a5384, State: Initialized, Role: FOLLOWER
I20260928 04:49:57.212216 11645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434da8180e9c4bf297c05b35166a5384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260928 04:49:57.212316 11645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:49:57.212349 11645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:49:57.212414 11645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:49:57.213037 11645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434da8180e9c4bf297c05b35166a5384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260928 04:49:57.213146 11645 leader_election.cc:302] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 434da8180e9c4bf297c05b35166a5384; no voters:
I20260928 04:49:57.213354 11645 leader_election.cc:288] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:49:57.213369 11650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:49:57.213587 11650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [term 1 LEADER]: Becoming Leader. State: Replica: 434da8180e9c4bf297c05b35166a5384, State: Running, Role: LEADER
I20260928 04:49:57.213662 11645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:49:57.213712 11650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434da8180e9c4bf297c05b35166a5384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260928 04:49:57.214094 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "434da8180e9c4bf297c05b35166a5384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434da8180e9c4bf297c05b35166a5384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } } }
I20260928 04:49:57.214249 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [sys.catalog]: This master's current role is: LEADER
I20260928 04:49:57.214416 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 434da8180e9c4bf297c05b35166a5384. Latest consensus state: current_term: 1 leader_uuid: "434da8180e9c4bf297c05b35166a5384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434da8180e9c4bf297c05b35166a5384" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } } }
I20260928 04:49:57.214509 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384 [sys.catalog]: This master's current role is: LEADER
I20260928 04:49:57.214694 11659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:49:57.215160 11659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:49:57.215410 11645 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 04:49:57.216508 11667 mvcc.cc:205] Tried to move back new op lower bound from 7334178804597014528 to 7334178804588101632. Current Snapshot: MvccSnapshot[applied={T|T < 7334178804597014528}]
I20260928 04:49:57.216950 11659 catalog_manager.cc:1347] Generated new cluster ID: 42912eaec173489791d215bc1ebf8c15
I20260928 04:49:57.216997 11659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:49:57 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:49:57.222415 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:57.212089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.0.98.62:36447)
0928 04:49:57.212304 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:57.212312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:57.212344 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:49:57.212455 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:57.212461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:57.212514 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:57.212689 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:57.213403 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:57.214774 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:57.215825 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:57.216390 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:57.216507 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:57.219141 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:57.219145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:57.219145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:49:57.219154 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:57.220813 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:57.221160 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:57.221162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:57.221162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:57.221259 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:57.221409 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:57.221410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:57.221410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:57.221645 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:57.221752 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:57.222143 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 04:49:57.222246 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260928 04:49:57.237563 11659 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:49:57.238399 11659 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:49:57.253476 11659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384: Generated new TSK 0
I20260928 04:49:57.253747 11659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:49:57.983923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:49:58.099761 11690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:58.100001 11690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:58.100030 11690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:58.101487 11690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:58.101577 11690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:58.101639 11690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:49:58.103564 11690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:41325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:58.103783 11690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:58.104038 11690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:58.109994 11696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:58.109994 11698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:58.110428 11695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:58.111382 11690 server_base.cc:1034] running on GCE node
I20260928 04:49:58.111653 11690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:58.111871 11690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:58.113034 11690 hybrid_clock.cc:648] HybridClock initialized: now 1790570998113003 us; error 42 us; skew 500 ppm
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:49:58.116600 11690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:49:58.117125 11690 webserver.cc:460] Webserver started at http://127.0.98.1:39481/ using document root <none> and password file <none>
I20260928 04:49:58.117383 11690 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:58.117439 11690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:58.117619 11690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:49:58.130861 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "a5b764a418514cc9931dc86419f5599f"
format_stamp: "Formatted at 2026-09-28 04:49:58 on dist-test-slave-jccs"
server_key: "1e89bd7899aba6d86ea3848915243c7c"
server_key_iv: "16a4ea23ce5bd618dce2b5e7ed2adbb7"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:58.131366 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a5b764a418514cc9931dc86419f5599f"
format_stamp: "Formatted at 2026-09-28 04:49:58 on dist-test-slave-jccs"
server_key: "1e89bd7899aba6d86ea3848915243c7c"
server_key_iv: "16a4ea23ce5bd618dce2b5e7ed2adbb7"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:58.136417 11690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.001s
I20260928 04:49:58.147895 11707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:58.148129 11690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:49:58.148239 11690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "a5b764a418514cc9931dc86419f5599f"
format_stamp: "Formatted at 2026-09-28 04:49:58 on dist-test-slave-jccs"
server_key: "1e89bd7899aba6d86ea3848915243c7c"
server_key_iv: "16a4ea23ce5bd618dce2b5e7ed2adbb7"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:58.148335 11690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:58.171722 11690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:58.174139 11690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:58.174351 11690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:58.174686 11690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:58.175134 11690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:58.175239 11690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:58.175335 11690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:58.175417 11690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:58.183391 11690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39213
I20260928 04:49:58.183893 11690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:49:58.186331 11820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39213 every 8 connection(s)
I20260928 04:49:58.192745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 11690
I20260928 04:49:58.192857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:49:58.193089 392 external_mini_cluster.cc:1428] Setting key 34a39752b3818cf24489aea33f0e1656
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:49:58.200489 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.186392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.1:49917)
0928 04:49:58.189300 (+ 2908us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:58.189352 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:58.189972 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:58.190357 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:58.190362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:58.190614 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:58.190899 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.190906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.191962 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.191963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.193195 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.193198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.193849 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.193850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.194724 (+ 874us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.195318 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:58.195328 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:58.196883 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:58.198938 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:58.198944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:58.198946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:58.199218 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:58.199459 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:58.199459 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:58.199459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:58.199576 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:58.199847 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:58.199852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:58.200016 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 04:49:58.200115 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 04:49:58.200269 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2864,"thread_start_us":75,"threads_started":1}
I20260928 04:49:58.201010 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.186694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49917 (local address 127.0.98.62:36447)
0928 04:49:58.186846 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:58.186848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:58.190108 (+ 3260us) server_negotiation.cc:407] Connection header received
0928 04:49:58.190141 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:58.190142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:58.190157 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:58.190221 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:58.191022 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.191824 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.193298 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.193727 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.193784 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.196987 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:58.196988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:58.196989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:49:58.196996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:58.198844 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:58.199308 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:58.199309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:58.199309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:58.199375 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:58.199640 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:58.199641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:58.199641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:58.199767 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:58.199824 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:58.200843 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0928 04:49:58.200922 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260928 04:49:58.201697 11821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260928 04:49:58.201784 11821 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:58.201946 11821 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260928 04:49:58.202467 11609 ts_manager.cc:194] Registered new tserver with Master: a5b764a418514cc9931dc86419f5599f (127.0.98.1:39213)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:49:58.203305 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:49:58.227967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 04:49:58.353438 11827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:58.353758 11827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:58.353830 11827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:58.355580 11827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:58.355688 11827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:58.355777 11827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:49:58.357959 11827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:41325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:58.358404 11827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:58.358696 11827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:58.364853 11834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:58.364845 11837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:58.365128 11835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:58.365996 11827 server_base.cc:1034] running on GCE node
I20260928 04:49:58.366309 11827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:58.366546 11827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:58.381978 11827 hybrid_clock.cc:648] HybridClock initialized: now 1790570998381952 us; error 47 us; skew 500 ppm
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:49:58.384483 11827 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:49:58.384954 11827 webserver.cc:460] Webserver started at http://127.0.98.2:39759/ using document root <none> and password file <none>
I20260928 04:49:58.385210 11827 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:58.385267 11827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:58.385398 11827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:49:58.397148 11827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "d89f4198d9fd47caa1c0c773f5920e76"
format_stamp: "Formatted at 2026-09-28 04:49:58 on dist-test-slave-jccs"
server_key: "6b6a3d5d07661320141148229e87cb7e"
server_key_iv: "28d04406b532ea54230a0c6e767d9886"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:58.397701 11827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d89f4198d9fd47caa1c0c773f5920e76"
format_stamp: "Formatted at 2026-09-28 04:49:58 on dist-test-slave-jccs"
server_key: "6b6a3d5d07661320141148229e87cb7e"
server_key_iv: "28d04406b532ea54230a0c6e767d9886"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:58.401595 11827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 04:49:58.419916 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:58.420094 11827 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260928 04:49:58.420173 11827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "d89f4198d9fd47caa1c0c773f5920e76"
format_stamp: "Formatted at 2026-09-28 04:49:58 on dist-test-slave-jccs"
server_key: "6b6a3d5d07661320141148229e87cb7e"
server_key_iv: "28d04406b532ea54230a0c6e767d9886"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:58.420275 11827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:58.433990 11827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:58.436326 11827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:58.436508 11827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:58.436769 11827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:58.437145 11827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:58.437269 11827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:58.437343 11827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:58.437403 11827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:58.445817 11827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46477
I20260928 04:49:58.446273 11827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:49:58.448016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 11827
I20260928 04:49:58.448125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:49:58.448326 392 external_mini_cluster.cc:1428] Setting key 414017772d4c390a3e3b6208b4ade154
I20260928 04:49:58.452539 11958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46477 every 8 connection(s)
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:49:58.464154 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.453980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46693 (local address 127.0.98.62:36447)
0928 04:49:58.454020 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:58.454022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:58.454200 (+ 178us) server_negotiation.cc:407] Connection header received
0928 04:49:58.454860 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:58.454861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:58.454873 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:58.454900 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:58.455701 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.456421 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.457845 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.458174 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.458271 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.460796 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:58.460797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:58.460797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:49:58.460805 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:58.462532 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:58.463027 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:58.463029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:58.463029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:58.463092 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:58.463386 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:58.463388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:58.463388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:58.463528 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:58.463571 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:58.463954 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 04:49:58.464045 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:49:58.464847 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.453725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.2:46693)
0928 04:49:58.454085 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:58.454127 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:58.454721 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:58.455079 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:58.455083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:58.455326 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:58.455591 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.455596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.456548 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.456549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.457749 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.457752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.458298 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.458299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.458439 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.459118 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:58.459128 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:58.460678 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:58.462621 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:58.462627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:58.462629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:58.462931 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:58.463167 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:58.463168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:58.463168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:58.463305 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:58.463616 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:58.463622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:58.463804 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 04:49:58.464508 (+ 704us) client_negotiation.cc:240] Negotiation successful
0928 04:49:58.464653 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:49:58.465531 11959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260928 04:49:58.465616 11959 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:58.465786 11959 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260928 04:49:58.466168 11609 ts_manager.cc:194] Registered new tserver with Master: d89f4198d9fd47caa1c0c773f5920e76 (127.0.98.2:46477)
I20260928 04:49:58.466645 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:49:58.492398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:49:58.607029 11964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:49:58.607409 11964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:49:58.607489 11964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:49:58.609064 11964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:49:58.609176 11964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:49:58.609299 11964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:49:58.611212 11964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:41325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:49:58.611743 11964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:49:58.612020 11964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:49:58.618373 11970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:58.618393 11969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:49:58.621352 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:49:58.622428 11964 server_base.cc:1034] running on GCE node
I20260928 04:49:58.622716 11964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:49:58.622973 11964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:49:58.626271 11964 hybrid_clock.cc:648] HybridClock initialized: now 1790570998626250 us; error 34 us; skew 500 ppm
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:49:58.629446 11964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:49:58.629920 11964 webserver.cc:460] Webserver started at http://127.0.98.3:38721/ using document root <none> and password file <none>
I20260928 04:49:58.630165 11964 fs_manager.cc:359] Metadata directory not provided
I20260928 04:49:58.630213 11964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:49:58.630347 11964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:49:58.643357 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "b679cd8d4b844bd6b45dd7cb64966bb7"
format_stamp: "Formatted at 2026-09-28 04:49:58 on dist-test-slave-jccs"
server_key: "dbcce08f00a66c5a5fba3c1aaaae5c93"
server_key_iv: "8e645cf7f11510702a406d23a92c8865"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:58.643867 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b679cd8d4b844bd6b45dd7cb64966bb7"
format_stamp: "Formatted at 2026-09-28 04:49:58 on dist-test-slave-jccs"
server_key: "dbcce08f00a66c5a5fba3c1aaaae5c93"
server_key_iv: "8e645cf7f11510702a406d23a92c8865"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:58.660344 11964 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.001s sys 0.004s
I20260928 04:49:58.674008 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:58.674261 11964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 04:49:58.674340 11964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "b679cd8d4b844bd6b45dd7cb64966bb7"
format_stamp: "Formatted at 2026-09-28 04:49:58 on dist-test-slave-jccs"
server_key: "dbcce08f00a66c5a5fba3c1aaaae5c93"
server_key_iv: "8e645cf7f11510702a406d23a92c8865"
server_key_version: "kuduclusterkey@0"
I20260928 04:49:58.674419 11964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:49:58.708436 11964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:49:58.711400 11964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:49:58.711577 11964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:49:58.711833 11964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:49:58.712215 11964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:49:58.712294 11964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:58.712359 11964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:49:58.712409 11964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:49:58.719401 11964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42673
I20260928 04:49:58.719871 11964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:49:58.721733 12092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42673 every 8 connection(s)
I20260928 04:49:58.723167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 11964
I20260928 04:49:58.723289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:49:58.723481 392 external_mini_cluster.cc:1428] Setting key f1e6caa52a8c467075901630808476b9
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:49:58.742990 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.722375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59943 (local address 127.0.98.62:36447)
0928 04:49:58.722429 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:58.722432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:58.728852 (+ 6420us) server_negotiation.cc:407] Connection header received
0928 04:49:58.729571 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:58.729572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:58.729583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:58.729610 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:58.730388 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.731119 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.732502 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.732802 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.732861 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.739525 (+ 6664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:58.739526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:58.739527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:49:58.739534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:58.741149 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:58.741643 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:58.741645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:58.741645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:58.741700 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:58.741977 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:58.741978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:58.741978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:58.742101 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:58.742140 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:58.742797 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 04:49:58.742883 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:49:58.743883 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.722363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.3:59943)
0928 04:49:58.728694 (+ 6331us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:58.728748 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:58.729446 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:58.729778 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:58.729783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:58.730013 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:49:58.730286 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.730291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.731225 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.731225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.732407 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.732410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.734162 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.734163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.735944 (+ 1781us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.736973 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:49:58.736983 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:49:58.739418 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:49:58.741270 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:58.741276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:58.741279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:58.741553 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:58.741764 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:49:58.741765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:49:58.741766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:49:58.741907 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:49:58.742468 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:49:58.742474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:49:58.742644 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 04:49:58.743491 (+ 847us) client_negotiation.cc:240] Negotiation successful
0928 04:49:58.743650 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6256,"thread_start_us":76,"threads_started":1}
I20260928 04:49:58.744737 12093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260928 04:49:58.744838 12093 heartbeater.cc:464] Registering TS with master...
I20260928 04:49:58.745028 12093 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260928 04:49:58.745939 11609 ts_manager.cc:194] Registered new tserver with Master: b679cd8d4b844bd6b45dd7cb64966bb7 (127.0.98.3:42673)
I20260928 04:49:58.746352 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59943
I20260928 04:49:58.756431 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:49:58.768119 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.758210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.0.98.62:36447)
0928 04:49:58.758255 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:58.758257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:58.760874 (+ 2617us) server_negotiation.cc:407] Connection header received
0928 04:49:58.760975 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:58.760976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:58.760987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:58.761017 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:49:58.761823 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.762521 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.763530 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.763826 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.763873 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.765380 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:49:58.765381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:49:58.765381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:49:58.765388 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:49:58.766973 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:58.767326 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:58.767327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:58.767327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:58.767378 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:49:58.767552 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:49:58.767553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:49:58.767553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:49:58.767663 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:49:58.767728 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:58.767957 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 04:49:58.768031 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:49:58.770249 11609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:49:58.772356 11609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:49:58.781819 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.776110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39213 (local address 127.0.0.1:35450)
0928 04:49:58.777032 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:58.777038 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:58.777072 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:58.777358 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:58.777361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:58.777363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:49:58.777426 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.777428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.778970 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.778971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.780254 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.780257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.781537 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.781538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.781617 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.781631 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 04:49:58.781692 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 04:49:58.781741 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":58,"threads_started":1}
I20260928 04:49:58.782126 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.776374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.0.98.1:39213)
0928 04:49:58.776661 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:58.776666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:58.777123 (+ 457us) server_negotiation.cc:407] Connection header received
0928 04:49:58.777160 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:58.777162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:58.777178 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:58.777279 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:49:58.777499 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.778884 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.780349 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.781382 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.781450 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.781504 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:58.782019 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 04:49:58.782060 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":59,"threads_started":1}
I20260928 04:49:58.782776 11755 tablet_service.cc:1467] Processing CreateTablet for tablet eec4b83cd07f4c90929484dcb806b509 (DEFAULT_TABLE table=test-table [id=c9ddf3dc44c9498eb93a7213a0b29fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:49:58.783149 11755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec4b83cd07f4c90929484dcb806b509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:49:58.789619 12111 tablet_bootstrap.cc:492] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f: Bootstrap starting.
I20260928 04:49:58.790377 12111 tablet_bootstrap.cc:654] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f: Neither blocks nor log segments found. Creating new log.
I20260928 04:49:58.790686 12111 log.cc:824] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:49:58.797785 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.775941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46477 (local address 127.0.0.1:52538)
0928 04:49:58.776933 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:58.776941 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:58.776970 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:58.789738 (+ 12768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:58.789738 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:58.789741 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:49:58.789781 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.789783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.795407 (+ 5624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.795408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.796512 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.796515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.797521 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.797523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.797585 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.797589 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:49:58.797661 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:49:58.797699 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":68,"threads_started":1}
I20260928 04:49:58.797823 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.776051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.0.98.2:46477)
0928 04:49:58.789292 (+ 13241us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:58.789297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:58.789307 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:49:58.789347 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:58.789349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:58.789365 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:58.789445 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:49:58.794166 (+ 4721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.795314 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.796607 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.797431 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.797515 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.797593 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:58.797675 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 04:49:58.797714 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13198,"thread_start_us":71,"threads_started":1}
I20260928 04:49:58.798382 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.775696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42673 (local address 127.0.0.1:45958)
0928 04:49:58.776737 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:58.776781 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:58.776843 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:58.787226 (+ 10383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:58.787227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:58.787230 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:49:58.787283 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.787284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.788580 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.788581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.795165 (+ 6584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.795169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.798177 (+ 3008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.798178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.798241 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.798245 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:49:58.798281 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 04:49:58.798317 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":74,"threads_started":1}
I20260928 04:49:58.799052 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.775797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.0.98.3:42673)
0928 04:49:58.787062 (+ 11265us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:58.787066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:58.787076 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:49:58.787112 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:58.787114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:58.787130 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:58.787349 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:49:58.787419 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.788483 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.795299 (+ 6816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.797407 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.798808 (+ 1401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.798909 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:58.798937 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 04:49:58.798975 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11229,"thread_start_us":113,"threads_started":1}
I20260928 04:49:58.799079 12111 tablet_bootstrap.cc:492] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f: No bootstrap required, opened a new log
I20260928 04:49:58.799155 12111 ts_tablet_manager.cc:1397] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260928 04:49:58.799117 11893 tablet_service.cc:1467] Processing CreateTablet for tablet eec4b83cd07f4c90929484dcb806b509 (DEFAULT_TABLE table=test-table [id=c9ddf3dc44c9498eb93a7213a0b29fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:49:58.799503 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec4b83cd07f4c90929484dcb806b509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:49:58.799778 12027 tablet_service.cc:1467] Processing CreateTablet for tablet eec4b83cd07f4c90929484dcb806b509 (DEFAULT_TABLE table=test-table [id=c9ddf3dc44c9498eb93a7213a0b29fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:49:58.800071 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec4b83cd07f4c90929484dcb806b509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:49:58.801955 12111 raft_consensus.cc:348] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 } } peers { permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 } } peers { permanent_uuid: "a5b764a418514cc9931dc86419f5599f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } }
I20260928 04:49:58.802093 12111 raft_consensus.cc:374] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:49:58.802112 12111 raft_consensus.cc:729] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b764a418514cc9931dc86419f5599f, State: Initialized, Role: FOLLOWER
I20260928 04:49:58.802227 12111 consensus_queue.cc:260] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 } } peers { permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 } } peers { permanent_uuid: "a5b764a418514cc9931dc86419f5599f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } }
I20260928 04:49:58.802474 12111 ts_tablet_manager.cc:1428] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 04:49:58.802518 11821 heartbeater.cc:503] Master 127.0.98.62:36447 was elected leader, sending a full tablet report...
I20260928 04:49:58.803824 12114 tablet_bootstrap.cc:492] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: Bootstrap starting.
I20260928 04:49:58.805080 12114 tablet_bootstrap.cc:654] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: Neither blocks nor log segments found. Creating new log.
I20260928 04:49:58.815076 12113 raft_consensus.cc:482] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:49:58.815194 12113 raft_consensus.cc:504] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 } } peers { permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 } } peers { permanent_uuid: "a5b764a418514cc9931dc86419f5599f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } }
I20260928 04:49:58.816000 12113 leader_election.cc:288] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b679cd8d4b844bd6b45dd7cb64966bb7 (127.0.98.3:42673), d89f4198d9fd47caa1c0c773f5920e76 (127.0.98.2:46477)
I20260928 04:49:58.816984 12114 log.cc:824] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: Log is configured to *not* fsync() on all Append() calls
I20260928 04:49:58.826705 12115 tablet_bootstrap.cc:492] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: Bootstrap starting.
I20260928 04:49:58.827736 12115 tablet_bootstrap.cc:654] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: Neither blocks nor log segments found. Creating new log.
I20260928 04:49:58.829515 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.815654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42673 (local address 127.0.98.1:44221)
0928 04:49:58.823929 (+ 8275us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:58.823941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:58.824153 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:58.824309 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:58.824310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:58.824322 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:49:58.824454 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.824456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.825391 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.825392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.826516 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.826519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.828567 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.828568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.828640 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.828644 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:49:58.829400 (+ 756us) client_negotiation.cc:240] Negotiation successful
0928 04:49:58.829439 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8254,"thread_start_us":77,"threads_started":1}
I20260928 04:49:58.829366 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.815698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44221 (local address 127.0.98.3:42673)
0928 04:49:58.819328 (+ 3630us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:58.819330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:58.823997 (+ 4667us) server_negotiation.cc:407] Connection header received
0928 04:49:58.824218 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:58.824219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:58.824227 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:58.824254 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:49:58.824562 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.825303 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.827890 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.828457 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.828501 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.828511 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:58.829264 (+ 753us) server_negotiation.cc:299] Negotiation successful
0928 04:49:58.829303 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3611,"thread_start_us":79,"threads_started":1}
I20260928 04:49:58.830084 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec4b83cd07f4c90929484dcb806b509" candidate_uuid: "a5b764a418514cc9931dc86419f5599f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" is_pre_election: true
W20260928 04:49:58.831297 11709 leader_election.cc:341] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b679cd8d4b844bd6b45dd7cb64966bb7 (127.0.98.3:42673): Illegal state: must be running to vote when last-logged opid is not known
I20260928 04:49:58.831729 12115 log.cc:824] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: Log is configured to *not* fsync() on all Append() calls
I20260928 04:49:58.834884 12114 tablet_bootstrap.cc:492] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: No bootstrap required, opened a new log
I20260928 04:49:58.834956 12114 ts_tablet_manager.cc:1397] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20260928 04:49:58.836776 12114 raft_consensus.cc:348] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 } } peers { permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 } } peers { permanent_uuid: "a5b764a418514cc9931dc86419f5599f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } }
I20260928 04:49:58.836951 12114 raft_consensus.cc:374] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:49:58.836979 12114 raft_consensus.cc:729] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d89f4198d9fd47caa1c0c773f5920e76, State: Initialized, Role: FOLLOWER
Sep 28 04:49:58 dist-test-slave-jccs krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790570998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:49:58.838002 12114 consensus_queue.cc:260] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 } } peers { permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 } } peers { permanent_uuid: "a5b764a418514cc9931dc86419f5599f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } }
I20260928 04:49:58.838336 12114 ts_tablet_manager.cc:1428] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 04:49:58.838485 11959 heartbeater.cc:503] Master 127.0.98.62:36447 was elected leader, sending a full tablet report...
I20260928 04:49:58.838862 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.815956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50347 (local address 127.0.98.2:46477)
0928 04:49:58.816008 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:58.816010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:58.824082 (+ 8072us) server_negotiation.cc:407] Connection header received
0928 04:49:58.824146 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:58.824147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:58.824160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:58.824220 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:49:58.825317 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.825945 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.837282 (+ 11337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.837828 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.837873 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.837881 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:58.838756 (+ 875us) server_negotiation.cc:299] Negotiation successful
0928 04:49:58.838801 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:49:58.839071 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.815889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46477 (local address 127.0.98.1:50347)
0928 04:49:58.824034 (+ 8145us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:58.824042 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:58.824094 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:58.825094 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:49:58.825095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:49:58.825097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:49:58.825223 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.825226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.826040 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.826040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.827074 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:49:58.827077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.838589 (+ 11512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.838590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:49:58.838670 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.838674 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:49:58.838939 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 04:49:58.838980 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8134,"thread_start_us":76,"threads_started":1}
I20260928 04:49:58.839305 11913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec4b83cd07f4c90929484dcb806b509" candidate_uuid: "a5b764a418514cc9931dc86419f5599f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d89f4198d9fd47caa1c0c773f5920e76" is_pre_election: true
I20260928 04:49:58.839435 11913 raft_consensus.cc:2454] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b764a418514cc9931dc86419f5599f in term 0.
I20260928 04:49:58.839629 11708 leader_election.cc:302] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5b764a418514cc9931dc86419f5599f, d89f4198d9fd47caa1c0c773f5920e76; no voters: b679cd8d4b844bd6b45dd7cb64966bb7
I20260928 04:49:58.839797 12113 raft_consensus.cc:2781] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:49:58.839845 12113 raft_consensus.cc:482] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:49:58.839872 12113 raft_consensus.cc:3037] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:49:58.840673 12113 raft_consensus.cc:504] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 } } peers { permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 } } peers { permanent_uuid: "a5b764a418514cc9931dc86419f5599f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } }
I20260928 04:49:58.840803 12113 leader_election.cc:288] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [CANDIDATE]: Term 1 election: Requested vote from peers b679cd8d4b844bd6b45dd7cb64966bb7 (127.0.98.3:42673), d89f4198d9fd47caa1c0c773f5920e76 (127.0.98.2:46477)
I20260928 04:49:58.840938 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec4b83cd07f4c90929484dcb806b509" candidate_uuid: "a5b764a418514cc9931dc86419f5599f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7"
I20260928 04:49:58.840957 11913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec4b83cd07f4c90929484dcb806b509" candidate_uuid: "a5b764a418514cc9931dc86419f5599f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d89f4198d9fd47caa1c0c773f5920e76"
I20260928 04:49:58.841002 11913 raft_consensus.cc:3037] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [term 0 FOLLOWER]: Advancing to term 1
W20260928 04:49:58.841168 11709 leader_election.cc:341] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b679cd8d4b844bd6b45dd7cb64966bb7 (127.0.98.3:42673): Illegal state: must be running to vote when last-logged opid is not known
I20260928 04:49:58.841785 11913 raft_consensus.cc:2454] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b764a418514cc9931dc86419f5599f in term 1.
I20260928 04:49:58.841945 11708 leader_election.cc:302] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5b764a418514cc9931dc86419f5599f, d89f4198d9fd47caa1c0c773f5920e76; no voters: b679cd8d4b844bd6b45dd7cb64966bb7
I20260928 04:49:58.842027 12113 raft_consensus.cc:2781] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:49:58.842142 12113 raft_consensus.cc:686] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [term 1 LEADER]: Becoming Leader. State: Replica: a5b764a418514cc9931dc86419f5599f, State: Running, Role: LEADER
I20260928 04:49:58.842217 12113 consensus_queue.cc:237] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 } } peers { permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 } } peers { permanent_uuid: "a5b764a418514cc9931dc86419f5599f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } }
I20260928 04:49:58.842967 11609 catalog_manager.cc:5495] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b764a418514cc9931dc86419f5599f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a5b764a418514cc9931dc86419f5599f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b764a418514cc9931dc86419f5599f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } health_report { overall_health: HEALTHY } } }
I20260928 04:49:58.843928 12115 tablet_bootstrap.cc:492] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: No bootstrap required, opened a new log
I20260928 04:49:58.844028 12115 ts_tablet_manager.cc:1397] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260928 04:49:58.845476 12115 raft_consensus.cc:348] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 } } peers { permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 } } peers { permanent_uuid: "a5b764a418514cc9931dc86419f5599f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } }
I20260928 04:49:58.845664 12115 raft_consensus.cc:374] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:49:58.845734 12115 raft_consensus.cc:729] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b679cd8d4b844bd6b45dd7cb64966bb7, State: Initialized, Role: FOLLOWER
I20260928 04:49:58.845911 12115 consensus_queue.cc:260] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 } } peers { permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 } } peers { permanent_uuid: "a5b764a418514cc9931dc86419f5599f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } }
I20260928 04:49:58.846222 12115 ts_tablet_manager.cc:1428] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:49:58.846370 12093 heartbeater.cc:503] Master 127.0.98.62:36447 was elected leader, sending a full tablet report...
I20260928 04:49:58.861455 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.858058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.0.98.1:39213)
0928 04:49:58.858120 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:49:58.858122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:49:58.858129 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:49:58.858166 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:49:58.858167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:49:58.858187 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:49:58.858253 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:49:58.858474 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.859227 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.860320 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:49:58.860610 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:49:58.860669 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:49:58.860871 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:49:58.861003 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:49:58.861081 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 04:49:58.861339 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 04:49:58.861359 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:49:58.879916 11913 raft_consensus.cc:1261] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [term 1 FOLLOWER]: Refusing update from remote peer a5b764a418514cc9931dc86419f5599f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:49:58.880349 12122 consensus_queue.cc:1035] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d89f4198d9fd47caa1c0c773f5920e76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:49:58.882529 12047 raft_consensus.cc:3037] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:49:58.883500 12047 raft_consensus.cc:1261] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7 [term 1 FOLLOWER]: Refusing update from remote peer a5b764a418514cc9931dc86419f5599f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:49:58.883852 12122 consensus_queue.cc:1035] T eec4b83cd07f4c90929484dcb806b509 P a5b764a418514cc9931dc86419f5599f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b679cd8d4b844bd6b45dd7cb64966bb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:49:58.886660 12126 mvcc.cc:205] Tried to move back new op lower bound from 7334178811408678912 to 7334178811258253312. Current Snapshot: MvccSnapshot[applied={T|T < 7334178811408678912}]
I20260928 04:49:58.887802 12129 mvcc.cc:205] Tried to move back new op lower bound from 7334178811408678912 to 7334178811258253312. Current Snapshot: MvccSnapshot[applied={T|T < 7334178811408678912}]
I20260928 04:49:58.890292 11609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:49:58.890378 11609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:49:58.891139 11609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 434da8180e9c4bf297c05b35166a5384: Sending DeleteTablet for 3 replicas of tablet eec4b83cd07f4c90929484dcb806b509
I20260928 04:49:58.891495 12027 tablet_service.cc:1514] Processing DeleteTablet for tablet eec4b83cd07f4c90929484dcb806b509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:49:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45958
I20260928 04:49:58.891547 11893 tablet_service.cc:1514] Processing DeleteTablet for tablet eec4b83cd07f4c90929484dcb806b509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:49:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52538
I20260928 04:49:58.891846 11755 tablet_service.cc:1514] Processing DeleteTablet for tablet eec4b83cd07f4c90929484dcb806b509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:49:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35450
I20260928 04:49:58.892432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 11690
I20260928 04:49:58.896667 12134 tablet_replica.cc:330] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: stopping tablet replica
I20260928 04:49:58.896852 12134 raft_consensus.cc:2229] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:49:58.896989 12134 raft_consensus.cc:2258] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:49:58.901162 12135 tablet_replica.cc:330] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: stopping tablet replica
I20260928 04:49:58.901356 12135 raft_consensus.cc:2229] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:49:58.901934 12134 ts_tablet_manager.cc:1905] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:49:58.903641 12135 pending_rounds.cc:62] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: Trying to abort 2 pending ops.
I20260928 04:49:58.903764 12135 pending_rounds.cc:68] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334178811408678912 op_type: WRITE_OP write_request { tablet_id: "eec4b83cd07f4c90929484dcb806b509" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\242\346\347\325\006\"8\n\ntest-admin\022*\n c9ddf3dc44c9498eb93a7213a0b29fcd\020\001\030\001 \001(\001" signature: "<redacted>""\215\234&@\263\203\003\300\033\355p\206\021l*D|\330G\275\323$|G\336\224\226\353M3M\275/\021\250$\256),\303\365\370\003\372\323\"\177%\005\tX\345a\234oB \013\002\270\370\234\n\326" signing_key_seq_num: 0 } } request_id { client_id: "27d8784d0def4ca9bc72dc36cf53be67" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260928 04:49:58.904210 12134 ts_tablet_manager.cc:1918] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:49:58.904352 12134 log.cc:1196] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/eec4b83cd07f4c90929484dcb806b509
I20260928 04:49:58.905032 12134 ts_tablet_manager.cc:1939] T eec4b83cd07f4c90929484dcb806b509 P b679cd8d4b844bd6b45dd7cb64966bb7: Deleting consensus metadata
I20260928 04:49:58.905478 11594 catalog_manager.cc:4841] TS b679cd8d4b844bd6b45dd7cb64966bb7 (127.0.98.3:42673): tablet eec4b83cd07f4c90929484dcb806b509 (table test-table [id=c9ddf3dc44c9498eb93a7213a0b29fcd]) successfully deleted
W20260928 04:49:58.903942 11846 connection.cc:441] server connection from 127.0.98.1:50347 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.98.1:50347 (request call id 3) could send its response
I20260928 04:49:58.905839 12135 pending_rounds.cc:68] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334178811258253312 op_type: NO_OP noop_request { }
I20260928 04:49:58.905961 12135 raft_consensus.cc:2866] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260928 04:49:58.906030 12135 raft_consensus.cc:2258] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:49:58.906442 12135 ts_tablet_manager.cc:1905] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 04:49:58.908239 11595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39213 (error 108)
I20260928 04:49:58.908938 12135 ts_tablet_manager.cc:1918] T eec4b83cd07f4c90929484dcb806b509 P d89f4198d9fd47caa1c0c773f5920e76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:49:58.909111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 11827
I20260928 04:49:58.916712 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:49:58.916582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39213 (local address 127.0.0.1:35476)
0928 04:49:58.916646 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:58.916679 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 04:49:58.916908 11595 catalog_manager.cc:4584] TS a5b764a418514cc9931dc86419f5599f (127.0.98.1:39213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eec4b83cd07f4c90929484dcb806b509: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39213: connect: Connection refused (error 111)
W20260928 04:49:58.917476 12107 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:49:58.917269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46477 (local address 127.0.0.1:52542)
0928 04:49:58.917300 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 04:49:58.917308 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:49:58.917340 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:49:58.917444 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46477: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 04:49:58.917902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 11964
I20260928 04:49:58.923902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 11577
I20260928 04:49:58.930172 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 04:49:59.275590 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 04:49:59.275684 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 04:49:59.620630 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 04:49:59.620 UTC [11163] LOG: received smart shutdown request
2026-09-28 04:49:59.624 UTC [11163] LOG: background worker "logical replication launcher" (PID 11172) exited with exit code 1
2026-09-28 04:49:59.624 UTC [11167] LOG: shutting down
2026-09-28 04:49:59.787 UTC [11163] LOG: database system is shut down
2026-09-28T04:49:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44965 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:49:59 dist-test-slave-jccs krb5kdc[12146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:49:59 dist-test-slave-jccs krb5kdc[12146](info): set up 2 sockets
Sep 28 04:49:59 dist-test-slave-jccs krb5kdc[12146](info): commencing operation
krb5kdc: starting...
W20260928 04:50:00.913087 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:50:00 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:50:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 04:50:01.069485 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNuqiey/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/postgres -l logfile start
2026-09-28 04:50:03.509 UTC [12176] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 04:50:03.510 UTC [12176] LOG: listening on IPv4 address "127.0.98.20", port 35027
2026-09-28 04:50:03.513 UTC [12176] LOG: listening on Unix socket "/tmp/.s.PGSQL.35027"
2026-09-28 04:50:03.533 UTC [12179] LOG: database system was shut down at 2026-09-28 04:50:02 UTC
2026-09-28 04:50:03.536 UTC [12176] LOG: database system is ready to accept connections
I20260928 04:50:04.484961 392 mini_postgres.cc:96] Postgres bound to 35027
2026-09-28 04:50:04.490 UTC [12187] FATAL: database "slave" does not exist
127.0.98.20:35027 - accepting connections
I20260928 04:50:04.491351 392 mini_ranger.cc:161] Starting Ranger...
I20260928 04:50:04.505322 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 04:50:04.987738 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 04:50:04.987828 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-admin
2026-09-28 04:50:05,266 [I] DB FLAVOR :POSTGRES
2026-09-28 04:50:05,266 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:50:05,266 [I] Checking connection..
2026-09-28 04:50:05,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 04:50:05,524 [I] Checking connection passed.
2026-09-28 04:50:05,524 [I] --------- Verifying version history table ---------
2026-09-28 04:50:05,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:50:05,756 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:50:05,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:50:06,012 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 04:50:06,013 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 04:50:06,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 04:50:06.245 UTC [12251] WARNING: there is no transaction in progress
2026-09-28 04:50:06,251 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 04:50:06,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 04:50:06,485 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 04:50:06,485 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:50:06,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 04:50:06,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 04:50:06,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jccs.c.gcp-upstream.internal','N') ;"
2026-09-28 04:50:07,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:50:07,472 [I] Table x_portal_user does not exist in database ranger
2026-09-28 04:50:07,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:50:07,731 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 04:50:07,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:50:07,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:50:08,241 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 04:50:08,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 04:50:09.179 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.194 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.211 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.226 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.242 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.261 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.316 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.323 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.337 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.347 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.358 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.371 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.381 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.387 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.394 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09.401 UTC [12359] WARNING: there is no transaction in progress
2026-09-28 04:50:09,997 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 04:50:09,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 04:50:10,237 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 04:50:10,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 04:50:10,502 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 04:50:10,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 04:50:10,747 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 04:50:10,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 04:50:11,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 04:50:11,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jccs.c.gcp-upstream.internal';"
2026-09-28 04:50:11,501 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 04:50:11,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 04:50:11,739 [I] DB_PATCHES have already been applied
I20260928 04:50:11.745402 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNuqiey/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 04:50:11.745462 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 04:50:11.748374 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:47827
Sep 28, 2026 4:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:50:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 04:50:12.855903 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.365s user 0.001s sys 0.008s
Sep 28, 2026 4:50:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:50:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 4:50:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:50:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:50:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 04:50:33.125886 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 04:50:33 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571033, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 04:50:33.217525 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 04:50:33.226055 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 04:50:34.037274 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 04:50:34.037354 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-kms
I20260928 04:50:34.037372 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:35027
2026-09-28 04:50:34,190 [I] DB FLAVOR :POSTGRES
2026-09-28 04:50:34,191 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 04:50:34,191 [I] Checking connection
2026-09-28 04:50:34,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 04:50:34,421 [I] connection success
2026-09-28 04:50:34,422 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 04:50:34,422 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 04:50:34,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 04:50:34,660 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 04:50:34,660 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 04:50:34,660 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 04:50:34,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 04:50:34,898 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 04:50:35.310973 392 mini_ranger_kms.cc:310] Created kms service
I20260928 04:50:35.469664 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 04:50:35.512368 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 04:50:35 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571033, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:50:35.912534 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 04:50:35.912740 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNuqiey/build/release/bin/postgresql.jar:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNuqiey/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 04:50:35.912767 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 04:50:35.916007 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12539
I20260928 04:50:35.916079 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51667
Sep 28, 2026 4:50:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:50:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:50:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:50:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:50:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:50:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 04:50:38.092108 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.875s user 0.008s sys 0.130s
I20260928 04:50:38.092216 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 04:50:38.092275 392 mini_ranger_kms.cc:401] 127.0.98.20:51667/kms/v1/keys
Sep 28 04:50:38 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790571038, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 04:50:38 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571038, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 04:50:38 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571038, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 04:50:40 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:50:41.024821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34739
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34739
--ranger_config_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51667/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:41.121276 12590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:41.121506 12590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:41.121543 12590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:41.122699 12590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:50:41.122756 12590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:41.122777 12590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:41.122804 12590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:50:41.122836 12590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:50:41.124332 12590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51667/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34739
--ranger_config_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:41.124588 12590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:41.124802 12590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:41.129523 12596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:41.129539 12595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:41.129746 12598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:41.129912 12590 server_base.cc:1034] running on GCE node
I20260928 04:50:41.130178 12590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:41.130436 12590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:41.131599 12590 hybrid_clock.cc:648] HybridClock initialized: now 1790571041131587 us; error 28 us; skew 500 ppm
Sep 28 04:50:41 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:41.137844 12590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:50:41.138351 12590 webserver.cc:460] Webserver started at http://127.0.98.62:44273/ using document root <none> and password file <none>
I20260928 04:50:41.138597 12590 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:41.138645 12590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:41.138772 12590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:50:41 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:50:41.188633 12590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "28da0e9942a84239b05b6136125620af"
format_stamp: "Formatted at 2026-09-28 04:50:41 on dist-test-slave-jccs"
server_key: "f4b576981522ea5cc362050e2790c968"
server_key_iv: "701f43efbe62bb5e885d9347bc564295"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:41.189034 12590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "28da0e9942a84239b05b6136125620af"
format_stamp: "Formatted at 2026-09-28 04:50:41 on dist-test-slave-jccs"
server_key: "f4b576981522ea5cc362050e2790c968"
server_key_iv: "701f43efbe62bb5e885d9347bc564295"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:41.190429 12590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:50:41.201388 12605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:41.201594 12590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:41.201684 12590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "28da0e9942a84239b05b6136125620af"
format_stamp: "Formatted at 2026-09-28 04:50:41 on dist-test-slave-jccs"
server_key: "f4b576981522ea5cc362050e2790c968"
server_key_iv: "701f43efbe62bb5e885d9347bc564295"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:41.201792 12590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:41.213493 12590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:41.215449 12590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:41.215610 12590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:41.220448 12590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34739
I20260928 04:50:41.220474 12657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34739 every 8 connection(s)
I20260928 04:50:41.220834 12590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:50:41.221621 12658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:41.223968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 12590
I20260928 04:50:41.224088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:50:41.224262 392 external_mini_cluster.cc:1428] Setting key de9f5cb23f08c076e9482f240dbae342
I20260928 04:50:41.224457 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: Bootstrap starting.
I20260928 04:50:41.225135 12658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:41.225474 12658 log.cc:824] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:41.226899 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: No bootstrap required, opened a new log
I20260928 04:50:41.229415 12658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } }
I20260928 04:50:41.229586 12658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:41.229619 12658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28da0e9942a84239b05b6136125620af, State: Initialized, Role: FOLLOWER
I20260928 04:50:41.229792 12658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } }
I20260928 04:50:41.229873 12658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:50:41.229902 12658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:50:41.229978 12658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:41.230634 12658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } }
I20260928 04:50:41.230739 12658 leader_election.cc:302] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28da0e9942a84239b05b6136125620af; no voters:
I20260928 04:50:41.230937 12658 leader_election.cc:288] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:50:41.230981 12663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 04:50:41 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:41.231632 12663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 1 LEADER]: Becoming Leader. State: Replica: 28da0e9942a84239b05b6136125620af, State: Running, Role: LEADER
I20260928 04:50:41.231752 12658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:50:41.231775 12663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } }
I20260928 04:50:41.232286 12663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28da0e9942a84239b05b6136125620af. Latest consensus state: current_term: 1 leader_uuid: "28da0e9942a84239b05b6136125620af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } } }
I20260928 04:50:41.232420 12663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:41.232664 12672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:50:41.233112 12665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28da0e9942a84239b05b6136125620af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } } }
I20260928 04:50:41.233294 12665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:41.234239 12672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:50:41.234499 12658 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 04:50:41.235793 12680 mvcc.cc:205] Tried to move back new op lower bound from 7334178984899538944 to 7334178984886157312. Current Snapshot: MvccSnapshot[applied={T|T < 7334178984899538944}]
I20260928 04:50:41.236196 12672 catalog_manager.cc:1347] Generated new cluster ID: b53b8ba905b644e8849261234c1f0454
I20260928 04:50:41.236246 12672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:50:41.239248 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:41.224771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.0.98.62:34739)
0928 04:50:41.224958 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:41.224969 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:41.225049 (+ 80us) server_negotiation.cc:407] Connection header received
0928 04:50:41.225612 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:41.225621 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:41.225707 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:41.225856 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:41.226609 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:41.227590 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:41.228368 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:41.228835 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:41.228954 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:41.231577 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:41.231582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:41.231583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:41.231596 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:41.233734 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:41.237263 (+ 3529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:41.237267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:41.237267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:41.237375 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:41.237608 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:41.237609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:41.237609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:41.237909 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:41.238842 (+ 933us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:41.238931 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 04:50:41.239046 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260928 04:50:41.246452 12672 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:50:41.247033 12672 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:50:41.250054 12672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: Generated new TSK 0
I20260928 04:50:41.250209 12672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:50:41.965467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34739
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:42.048646 12703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:42.048864 12703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:42.048903 12703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:42.050160 12703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:42.050223 12703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:42.050266 12703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:50:42.051774 12703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:42.052026 12703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:42.052249 12703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:42.056982 12711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:42.056949 12709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:42.057196 12708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:42.057662 12703 server_base.cc:1034] running on GCE node
I20260928 04:50:42.057873 12703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:42.058135 12703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:42.059286 12703 hybrid_clock.cc:648] HybridClock initialized: now 1790571042059265 us; error 39 us; skew 500 ppm
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:42.061915 12703 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:50:42.062304 12703 webserver.cc:460] Webserver started at http://127.0.98.1:38281/ using document root <none> and password file <none>
I20260928 04:50:42.062533 12703 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:42.062615 12703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:42.062763 12703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:50:42.076200 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "5184c234755943dfaed58a03c1f828e4"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "b4d1ede60cd290bff13a50be0f83da1c"
server_key_iv: "b2e26bacbd1eba29b0ca8717f4319b6c"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.076783 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5184c234755943dfaed58a03c1f828e4"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "b4d1ede60cd290bff13a50be0f83da1c"
server_key_iv: "b2e26bacbd1eba29b0ca8717f4319b6c"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.079916 12703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 04:50:42.100262 12718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.100456 12703 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260928 04:50:42.100545 12703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "5184c234755943dfaed58a03c1f828e4"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "b4d1ede60cd290bff13a50be0f83da1c"
server_key_iv: "b2e26bacbd1eba29b0ca8717f4319b6c"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.100668 12703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:42.135208 12703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:42.137666 12703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:42.137852 12703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:42.138111 12703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:42.138518 12703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:42.138602 12703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.138720 12703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:42.138799 12703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.145710 12703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34007
I20260928 04:50:42.145793 12833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34007 every 8 connection(s)
I20260928 04:50:42.146178 12703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:50:42.150978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 12703
I20260928 04:50:42.151101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:50:42.151274 392 external_mini_cluster.cc:1428] Setting key 9efbc7cc26f8ba95db107a9425a9f036
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:42.158397 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:42.146976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34739 (local address 127.0.98.1:58121)
0928 04:50:42.147166 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:42.147219 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:42.147672 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:42.148027 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:42.148032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:42.148213 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:42.148399 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:42.148403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.149133 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.149134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:42.149966 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:42.149969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.150401 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.150402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:42.150524 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:42.151150 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:42.151161 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:42.154808 (+ 3647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:42.156798 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:42.156805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:42.156807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:42.157033 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:42.157291 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:42.157291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:42.157292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:42.157425 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:42.157722 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:42.157728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:42.157915 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 04:50:42.158006 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 04:50:42.158171 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260928 04:50:42.159217 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:42.147004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58121 (local address 127.0.98.62:34739)
0928 04:50:42.147816 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:42.147818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:42.147824 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:50:42.147846 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:42.147847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:42.147856 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:42.147937 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:42.148497 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.149023 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.150069 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.150289 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.150375 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:42.154914 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:42.154916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:42.154916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:42.154922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:42.156704 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:42.157122 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:42.157123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:42.157124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:42.157210 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:42.157504 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:42.157505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:42.157505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:42.157638 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:42.158973 (+ 1335us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:42.159016 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 04:50:42.159105 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":112,"threads_started":1}
I20260928 04:50:42.159777 12834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34739
I20260928 04:50:42.159879 12834 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:42.160058 12834 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
I20260928 04:50:42.160674 12622 ts_manager.cc:194] Registered new tserver with Master: 5184c234755943dfaed58a03c1f828e4 (127.0.98.1:34007)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 04:50:42.161614 12622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:50:42.180675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34739
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:42.281085 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:42.281376 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:42.281418 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:42.282701 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:42.282769 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:42.282814 12840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:50:42.284247 12840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:42.284505 12840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:42.284756 12840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:42.289713 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:42.289690 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:42.290051 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:42.290410 12840 server_base.cc:1034] running on GCE node
I20260928 04:50:42.290613 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:42.290876 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:42.292032 12840 hybrid_clock.cc:648] HybridClock initialized: now 1790571042292020 us; error 29 us; skew 500 ppm
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:42.294831 12840 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:50:42.295284 12840 webserver.cc:460] Webserver started at http://127.0.98.2:37321/ using document root <none> and password file <none>
I20260928 04:50:42.295641 12840 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:42.295744 12840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:42.295899 12840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:42.317211 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "b97dd89cbf8a40d28fc288faa24e8c15"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "aea2f0a550e81ed521aa118a65a9a307"
server_key_iv: "53f3c511b9df4356e3a4b680e9ef6f3a"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.317800 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b97dd89cbf8a40d28fc288faa24e8c15"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "aea2f0a550e81ed521aa118a65a9a307"
server_key_iv: "53f3c511b9df4356e3a4b680e9ef6f3a"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.320744 12840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 04:50:42.340404 12857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.340682 12840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:42.340780 12840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "b97dd89cbf8a40d28fc288faa24e8c15"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "aea2f0a550e81ed521aa118a65a9a307"
server_key_iv: "53f3c511b9df4356e3a4b680e9ef6f3a"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.340878 12840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:42.357054 12840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:42.359447 12840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:42.359634 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:42.359879 12840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:42.360270 12840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:42.360327 12840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.360391 12840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:42.360420 12840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.367774 12840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34979
I20260928 04:50:42.368296 12840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:50:42.373514 12970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34979 every 8 connection(s)
I20260928 04:50:42.376518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 12840
I20260928 04:50:42.376621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:50:42.376833 392 external_mini_cluster.cc:1428] Setting key 8488da8f7ac234ff0b803ba04f83892d
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:42.383868 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:42.370695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47103 (local address 127.0.98.62:34739)
0928 04:50:42.370739 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:42.370741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:42.371050 (+ 309us) server_negotiation.cc:407] Connection header received
0928 04:50:42.371747 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:42.371749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:42.371760 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:42.371790 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:42.372555 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.373384 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.374920 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.375233 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.375327 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:42.380286 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:42.380288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:42.380288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:42.380296 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:42.382273 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:42.382770 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:42.382771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:42.382772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:42.382830 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:42.383116 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:42.383117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:42.383117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:42.383267 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:42.383310 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:42.383701 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 04:50:42.383782 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:50:42.384371 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:42.370446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34739 (local address 127.0.98.2:47103)
0928 04:50:42.370935 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:42.370975 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:42.371609 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:42.371968 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:42.371973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:42.372197 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:42.372459 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:42.372463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.373510 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.373511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:42.374807 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:42.374809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.375356 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.375356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:42.375483 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:42.376080 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:42.376089 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:42.380141 (+ 4052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:42.382372 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:42.382378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:42.382381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:42.382673 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:42.382904 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:42.382905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:42.382905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:42.383031 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:42.383359 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:42.383365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:42.383538 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 04:50:42.383977 (+ 439us) client_negotiation.cc:240] Negotiation successful
0928 04:50:42.384129 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":71,"threads_started":1}
I20260928 04:50:42.385020 12971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34739
I20260928 04:50:42.385116 12971 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:42.385314 12971 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
I20260928 04:50:42.385648 12622 ts_manager.cc:194] Registered new tserver with Master: b97dd89cbf8a40d28fc288faa24e8c15 (127.0.98.2:34979)
I20260928 04:50:42.386152 12622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47103
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:50:42.406538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34739
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:42.527668 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:42.527994 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:42.528031 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:42.529644 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:42.529760 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:42.529832 12976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:50:42.531917 12976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:42.532168 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:42.532419 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:42.537349 12985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:42.537335 12982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:42.537339 12983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:42.537752 12976 server_base.cc:1034] running on GCE node
I20260928 04:50:42.537971 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:42.538242 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:42.539438 12976 hybrid_clock.cc:648] HybridClock initialized: now 1790571042539423 us; error 45 us; skew 500 ppm
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:42.542698 12976 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:50:42.543224 12976 webserver.cc:460] Webserver started at http://127.0.98.3:44563/ using document root <none> and password file <none>
I20260928 04:50:42.543471 12976 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:42.543546 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:42.543685 12976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:50:42.556888 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "dfbaaf4736ae41579115c6e8f452e87f"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "9e0870523d58a89291a69a6a21384cd2"
server_key_iv: "3945ecf077e40ce71805dbb70704dd54"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.557358 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dfbaaf4736ae41579115c6e8f452e87f"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "9e0870523d58a89291a69a6a21384cd2"
server_key_iv: "3945ecf077e40ce71805dbb70704dd54"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.559536 12976 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 04:50:42.577571 12992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.577834 12976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 04:50:42.577931 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "dfbaaf4736ae41579115c6e8f452e87f"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "9e0870523d58a89291a69a6a21384cd2"
server_key_iv: "3945ecf077e40ce71805dbb70704dd54"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.578053 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:42.616583 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:42.619096 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:42.619256 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:42.619463 12976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:42.619863 12976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:42.619908 12976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.619938 12976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:42.619961 12976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.627074 12976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33081
I20260928 04:50:42.627511 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:50:42.629341 13105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33081 every 8 connection(s)
I20260928 04:50:42.635367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 12976
I20260928 04:50:42.635479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:50:42.635697 392 external_mini_cluster.cc:1428] Setting key b4225a78177282b8bb8cb0400b1266f8
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:42.639935 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:42.630031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40959 (local address 127.0.98.62:34739)
0928 04:50:42.630075 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:42.630077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:42.630340 (+ 263us) server_negotiation.cc:407] Connection header received
0928 04:50:42.630964 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:42.630965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:42.630978 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:42.631050 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:42.631830 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.632552 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.633895 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.634145 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.634183 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:42.636906 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:42.636908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:42.636908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:42.636915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:42.638540 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:42.638961 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:42.638962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:42.638963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:42.639016 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:42.639269 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:42.639270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:42.639271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:42.639389 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:42.639464 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:42.639770 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 04:50:42.639848 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:50:42.640118 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:42.630027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34739 (local address 127.0.98.3:40959)
0928 04:50:42.630177 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:42.630231 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:42.630846 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:42.631187 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:42.631191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:42.631453 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:42.631745 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:42.631751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.632639 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.632640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:42.633800 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:42.633802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:42.634297 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:42.634298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:42.634421 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:42.635067 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:42.635077 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:42.636803 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:42.638628 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:42.638634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:42.638636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:42.638858 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:42.639090 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:42.639091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:42.639091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:42.639202 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:42.639455 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:42.639461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:42.639608 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 04:50:42.639738 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 04:50:42.639878 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260928 04:50:42.640568 13106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34739
I20260928 04:50:42.640662 13106 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:42.640841 13106 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
I20260928 04:50:42.641254 12622 ts_manager.cc:194] Registered new tserver with Master: dfbaaf4736ae41579115c6e8f452e87f (127.0.98.3:33081)
I20260928 04:50:42.641628 12622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40959
I20260928 04:50:42.647781 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:50:42.648442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 12703
I20260928 04:50:42.654795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 12840
I20260928 04:50:42.660835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 12976
I20260928 04:50:42.665400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 12590
I20260928 04:50:42.670024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34739
--webserver_interface=127.0.98.62
--webserver_port=44273
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34739
--ranger_config_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51667/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51667/kms with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:42.748819 13109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:42.748996 13109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:42.749015 13109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:42.749990 13109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:50:42.750021 13109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:42.750034 13109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:42.750046 13109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:50:42.750056 13109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:50:42.751430 13109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51667/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34739
--ranger_config_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:42.751595 13109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:42.751751 13109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:42.756361 13117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:42.756358 13115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:42.756505 13109 server_base.cc:1034] running on GCE node
W20260928 04:50:42.756366 13114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:42.756815 13109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:42.757133 13109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:42.758308 13109 hybrid_clock.cc:648] HybridClock initialized: now 1790571042758292 us; error 37 us; skew 500 ppm
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:42.760694 13109 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:50:42.761063 13109 webserver.cc:460] Webserver started at http://127.0.98.62:44273/ using document root <none> and password file <none>
I20260928 04:50:42.761289 13109 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:42.761341 13109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:50:42.824424 13109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:42.825218 13125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.825440 13109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:42.825531 13109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "28da0e9942a84239b05b6136125620af"
format_stamp: "Formatted at 2026-09-28 04:50:41 on dist-test-slave-jccs"
server_key: "f4b576981522ea5cc362050e2790c968"
server_key_iv: "701f43efbe62bb5e885d9347bc564295"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.825904 13109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:42.843595 13109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:42.845528 13109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:42.845726 13109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:42.850030 13109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34739
I20260928 04:50:42.850065 13177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34739 every 8 connection(s)
I20260928 04:50:42.850410 13109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:50:42.851567 13178 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:50:42.852231 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: Bootstrap starting.
I20260928 04:50:42.855912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 13109
I20260928 04:50:42.856204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34007
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34739
--webserver_port=38281
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51667/kms with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:50:42.860054 13178 log.cc:824] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:42.861791 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:50:42.862057 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: Bootstrap complete.
I20260928 04:50:42.864352 13178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } }
I20260928 04:50:42.864585 13178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28da0e9942a84239b05b6136125620af, State: Initialized, Role: FOLLOWER
I20260928 04:50:42.864784 13178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } }
I20260928 04:50:42.864876 13178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:50:42.864925 13178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:50:42.864972 13178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:50:42.865975 13178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } }
I20260928 04:50:42.866086 13178 leader_election.cc:302] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28da0e9942a84239b05b6136125620af; no voters:
I20260928 04:50:42.866240 13178 leader_election.cc:288] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:50:42.866287 13182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:50:42.866418 13182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [term 2 LEADER]: Becoming Leader. State: Replica: 28da0e9942a84239b05b6136125620af, State: Running, Role: LEADER
I20260928 04:50:42.866485 13178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:50:42.866503 13182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } }
I20260928 04:50:42.866745 13184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "28da0e9942a84239b05b6136125620af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } } }
I20260928 04:50:42.866744 13182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28da0e9942a84239b05b6136125620af. Latest consensus state: current_term: 2 leader_uuid: "28da0e9942a84239b05b6136125620af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28da0e9942a84239b05b6136125620af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34739 } } }
I20260928 04:50:42.866874 13184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:42.866881 13182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:42.867954 13178 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 04:50:42.869931 13187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:50:42.870858 13187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:50:42.871063 13187 catalog_manager.cc:1259] Loaded cluster ID: b53b8ba905b644e8849261234c1f0454
I20260928 04:50:42.871105 13187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:50:42.873000 13187 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:50:42.873246 13187 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: Loaded TSK: 0
I20260928 04:50:42.873494 13187 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:50:42.940524 13180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:42.940841 13180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:42.940886 13180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:42.942528 13180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:42.942579 13180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:42.942618 13180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:50:42.944687 13180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38281
--tserver_master_addrs=127.0.98.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:42.944939 13180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:42.945199 13180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:42.950134 13216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:42.950182 13218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:42.950512 13215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:42.951048 13180 server_base.cc:1034] running on GCE node
I20260928 04:50:42.951236 13180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:42.951504 13180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:42.954762 13180 hybrid_clock.cc:648] HybridClock initialized: now 1790571042954761 us; error 44 us; skew 500 ppm
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:42.957520 13180 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:50:42.957962 13180 webserver.cc:460] Webserver started at http://127.0.98.1:38281/ using document root <none> and password file <none>
I20260928 04:50:42.958233 13180 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:42.958314 13180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:50:42 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:50:42.982897 13180 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260928 04:50:42.985342 13226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:42.985497 13180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 04:50:42.985584 13180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "5184c234755943dfaed58a03c1f828e4"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "b4d1ede60cd290bff13a50be0f83da1c"
server_key_iv: "b2e26bacbd1eba29b0ca8717f4319b6c"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:42.986039 13180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:43.027295 13180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:43.029829 13180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:43.029978 13180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:43.030190 13180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:43.030543 13180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:43.030580 13180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:43.030608 13180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:43.030627 13180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:43.039991 13180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34007
I20260928 04:50:43.041100 13180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:50:43.043167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 13180
I20260928 04:50:43.043473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34979
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34739
--webserver_port=37321
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51667/kms with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:50:43.046496 13339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34007 every 8 connection(s)
Sep 28 04:50:43 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:43.056625 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:43.042637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34739 (local address 127.0.98.1:46095)
0928 04:50:43.043513 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:43.043556 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:43.044276 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:43.044931 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:43.044934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:43.045212 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:43.045511 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:43.045518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.047210 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.047211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:43.048551 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:43.048554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.049586 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.049587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:43.049723 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:43.050314 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:43.050325 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:43.052140 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:43.054611 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:43.054618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:43.054620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:43.054954 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:43.055246 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:43.055247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:43.055247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:43.055382 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:43.055893 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:43.055899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:43.056069 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 04:50:43.056181 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 04:50:43.056347 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":569,"threads_started":1}
I20260928 04:50:43.057452 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:43.042432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46095 (local address 127.0.98.62:34739)
0928 04:50:43.043423 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:43.043432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:43.044424 (+ 992us) server_negotiation.cc:407] Connection header received
0928 04:50:43.044566 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:43.044572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:43.044629 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:43.044772 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:43.045661 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.047103 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.048647 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.049368 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.049481 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:43.052266 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:43.052272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:43.052272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:43.052284 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:43.054515 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:43.055051 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:43.055054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:43.055054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:43.055165 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:43.055462 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:43.055463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:43.055464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:43.055779 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:43.055862 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:43.057118 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0928 04:50:43.057257 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":83,"threads_started":1}
I20260928 04:50:43.058518 13340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34739
I20260928 04:50:43.058604 13340 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:43.058785 13340 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
W20260928 04:50:43.059609 13340 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34739 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 04:50:43.059672 13340 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:43.059729 13340 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
I20260928 04:50:43.060084 13340 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:43.060150 13340 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
W20260928 04:50:43.060446 13340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:50:43.152920 13344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:43.153165 13344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:43.153221 13344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:43.154379 13344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:43.154462 13344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:43.154513 13344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:50:43.156076 13344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37321
--tserver_master_addrs=127.0.98.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:43.156339 13344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:43.156555 13344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:43.161509 13352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:43.161624 13350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:43.161509 13349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:43.162652 13344 server_base.cc:1034] running on GCE node
I20260928 04:50:43.162870 13344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:43.163156 13344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:43.164932 13344 hybrid_clock.cc:648] HybridClock initialized: now 1790571043164906 us; error 36 us; skew 500 ppm
Sep 28 04:50:43 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:43.168466 13344 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:50:43.168977 13344 webserver.cc:460] Webserver started at http://127.0.98.2:37321/ using document root <none> and password file <none>
I20260928 04:50:43.169265 13344 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:43.169340 13344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:50:43 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:50:43.187680 13344 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:43.188412 13360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:43.188572 13344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:43.188653 13344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "b97dd89cbf8a40d28fc288faa24e8c15"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "aea2f0a550e81ed521aa118a65a9a307"
server_key_iv: "53f3c511b9df4356e3a4b680e9ef6f3a"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:43.189015 13344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:43.217479 13344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:43.219946 13344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:43.220191 13344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:43.220460 13344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:43.220916 13344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:43.220963 13344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:43.220994 13344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:43.221014 13344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:43.227658 13344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34979
I20260928 04:50:43.228134 13344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:50:43.229125 13473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34979 every 8 connection(s)
I20260928 04:50:43.230818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 13344
I20260928 04:50:43.231141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51667/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33081
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34739
--webserver_port=44563
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51667/kms with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:50:43 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:43.248202 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:43.230062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33721 (local address 127.0.98.62:34739)
0928 04:50:43.230106 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:43.230109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:43.237559 (+ 7450us) server_negotiation.cc:407] Connection header received
0928 04:50:43.238426 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:43.238428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:43.238447 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:43.238482 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:43.239417 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.240299 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.241961 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.242287 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.242330 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:43.244850 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:43.244851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:43.244851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:43.244858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:43.246497 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:43.246969 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:43.246971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:43.246971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:43.247023 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:43.247462 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:43.247463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:43.247463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:43.247594 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:43.247635 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:43.247991 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 04:50:43.248089 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:50:43.248999 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:43.229795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34739 (local address 127.0.98.2:33721)
0928 04:50:43.237392 (+ 7597us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:43.237448 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:43.238281 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:43.238685 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:43.238690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:43.238959 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:43.239291 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:43.239298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.240422 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.240423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:43.241851 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:43.241854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.242427 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.242428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:43.242558 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:43.243167 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:43.243176 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:43.244747 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:43.246584 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:43.246590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:43.246593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:43.246886 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:43.247089 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:43.247089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:43.247090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:43.247385 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:43.247680 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:43.247685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:43.247851 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 04:50:43.248548 (+ 697us) client_negotiation.cc:240] Negotiation successful
0928 04:50:43.248693 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7550,"thread_start_us":78,"threads_started":1}
I20260928 04:50:43.249693 13474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34739
I20260928 04:50:43.249782 13474 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:43.249953 13474 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
W20260928 04:50:43.250556 13474 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34739 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 04:50:43.250635 13474 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:43.250689 13474 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
I20260928 04:50:43.251003 13474 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:43.251063 13474 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
W20260928 04:50:43.251336 13474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:50:43.331625 13477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:43.331905 13477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:43.331938 13477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:43.333630 13477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:43.333691 13477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:43.333724 13477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:50:43.335695 13477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51667/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44563
--tserver_master_addrs=127.0.98.62:34739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:43.335922 13477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:43.336146 13477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:43.342170 13483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:43.342262 13482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:43.342417 13477 server_base.cc:1034] running on GCE node
W20260928 04:50:43.342154 13485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:43.342705 13477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:43.342981 13477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:43.344169 13477 hybrid_clock.cc:648] HybridClock initialized: now 1790571043344140 us; error 44 us; skew 500 ppm
Sep 28 04:50:43 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:43.347464 13477 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:50:43.347980 13477 webserver.cc:460] Webserver started at http://127.0.98.3:44563/ using document root <none> and password file <none>
I20260928 04:50:43.348279 13477 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:43.348358 13477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 04:50:43 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:50:43.377604 13477 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.001s sys 0.000s
I20260928 04:50:43.378557 13494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:43.378749 13477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:43.378839 13477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "dfbaaf4736ae41579115c6e8f452e87f"
format_stamp: "Formatted at 2026-09-28 04:50:42 on dist-test-slave-jccs"
server_key: "9e0870523d58a89291a69a6a21384cd2"
server_key_iv: "3945ecf077e40ce71805dbb70704dd54"
server_key_version: "kuduclusterkey@0"
I20260928 04:50:43.379281 13477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:43.396821 13477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:43.398898 13477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:43.399045 13477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:43.399257 13477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:43.399578 13477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:43.399634 13477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:43.399672 13477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:43.399698 13477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:43.406047 13477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33081
I20260928 04:50:43.406116 13607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33081 every 8 connection(s)
I20260928 04:50:43.406459 13477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:50:43.408787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 13477
Sep 28 04:50:43 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:43.415985 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:43.407213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41947 (local address 127.0.98.62:34739)
0928 04:50:43.407258 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:43.407260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:43.407518 (+ 258us) server_negotiation.cc:407] Connection header received
0928 04:50:43.408058 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:43.408059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:43.408072 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:43.408171 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:43.408738 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.409494 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.410776 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.411076 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.411150 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:43.413237 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:43.413239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:43.413239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:43.413244 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:43.414664 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:43.415040 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:43.415042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:43.415043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:43.415093 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:43.415353 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:43.415355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:43.415355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:43.415480 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:43.415540 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:43.415830 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 04:50:43.415899 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:50:43.416074 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:43.407225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34739 (local address 127.0.98.3:41947)
0928 04:50:43.407380 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:43.407416 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:43.407934 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:43.408223 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:43.408226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:43.408401 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:43.408626 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:43.408632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.409713 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.409714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:43.410696 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:43.410699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.411180 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.411181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:43.411285 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:43.411813 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:43.411829 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:43.413111 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:43.414753 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:43.414759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:43.414761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:43.414959 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:43.415162 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:43.415163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:43.415163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:43.415284 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:43.415532 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:43.415538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:43.415704 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 04:50:43.415795 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 04:50:43.415894 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260928 04:50:43.416743 13608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34739
I20260928 04:50:43.416846 13608 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:43.417074 13608 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
W20260928 04:50:43.417768 13608 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34739 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 04:50:43.417850 13608 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:43.417924 13608 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
I20260928 04:50:43.418241 13608 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:43.418323 13608 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
W20260928 04:50:43.418645 13608 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:50:43.419066 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:43.409572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.0.98.62:34739)
0928 04:50:43.409679 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:43.409681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:43.409688 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:50:43.409708 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:43.409709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:43.409718 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:43.409783 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:43.410507 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.411038 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.412212 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:43.413078 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:43.413148 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:43.415232 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:43.415232 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:43.415233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:43.415238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:43.417381 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:43.417845 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:43.417846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:43.417846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:43.417908 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:43.418239 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:43.418240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:43.418240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:43.418369 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:43.418804 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:43.418850 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 04:50:43.418938 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":64,"threads_started":1}
Sep 28 04:50:43 dist-test-slave-jccs krb5kdc[12146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790571043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:44.060621 13340 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:44.060810 13340 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
I20260928 04:50:44.061223 13142 ts_manager.cc:194] Registered new tserver with Master: 5184c234755943dfaed58a03c1f828e4 (127.0.98.1:34007)
I20260928 04:50:44.062224 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46095
I20260928 04:50:44.251523 13474 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:44.251660 13474 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
I20260928 04:50:44.255283 13142 ts_manager.cc:194] Registered new tserver with Master: b97dd89cbf8a40d28fc288faa24e8c15 (127.0.98.2:34979)
I20260928 04:50:44.255815 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33721
Sep 28 04:50:44 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:50:44.418864 13608 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:44.419018 13608 heartbeater.cc:511] Master 127.0.98.62:34739 requested a full tablet report, sending...
I20260928 04:50:44.419364 13142 ts_manager.cc:194] Registered new tserver with Master: dfbaaf4736ae41579115c6e8f452e87f (127.0.98.3:33081)
I20260928 04:50:44.419884 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41947
I20260928 04:50:44.423430 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:50:44.431770 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.424420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.0.98.62:34739)
0928 04:50:44.424653 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:44.424655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:44.424662 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:50:44.424681 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:44.424682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:44.424692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:44.424757 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:44.425385 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.426110 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.426990 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.427313 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.427387 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.428873 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:44.428875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:44.428875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:44.428882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:44.430662 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:44.431032 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:44.431033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:44.431033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:44.431092 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:44.431277 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:44.431278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:44.431278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:44.431409 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:44.431502 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:44.431598 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 04:50:44.431672 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":65,"threads_started":1}
I20260928 04:50:44.434031 13142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:50:44.436316 13142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:50:44.450038 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.443310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33081 (local address 127.0.0.1:38546)
0928 04:50:44.445286 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:44.445296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:44.445340 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:44.445579 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:44.445582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:44.445584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:44.445656 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:44.445658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.447225 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.447226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:44.448781 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:44.448784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.449738 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.449739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:44.449827 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.449841 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 04:50:44.449911 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:50:44.449959 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1961,"thread_start_us":313,"threads_started":1}
I20260928 04:50:44.450178 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.443306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34979 (local address 127.0.0.1:47320)
0928 04:50:44.443911 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:44.443947 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:44.444071 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:44.445729 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:44.445729 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:44.445731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:44.445752 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:44.445753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.448145 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.448146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:44.449088 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:44.449090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.449996 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.449997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:44.450052 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.450056 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:50:44.450080 (+ 24us) client_negotiation.cc:240] Negotiation successful
0928 04:50:44.450112 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":316,"threads_started":1}
I20260928 04:50:44.450594 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.443458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.0.98.2:34979)
0928 04:50:44.445436 (+ 1978us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:44.445441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:44.445450 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:50:44.445483 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:44.445484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:44.445499 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:44.445804 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:44.445872 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.448049 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.449194 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.449910 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.450241 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.450314 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:44.450489 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 04:50:44.450527 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1947,"thread_start_us":49,"threads_started":1}
I20260928 04:50:44.450855 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.443449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.0.98.3:33081)
0928 04:50:44.445329 (+ 1880us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:44.445334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:44.445349 (+ 15us) server_negotiation.cc:407] Connection header received
0928 04:50:44.445392 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:44.445394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:44.445410 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:44.445499 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:44.446162 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.447100 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.448868 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.449545 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.449620 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.449694 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:44.450626 (+ 932us) server_negotiation.cc:299] Negotiation successful
0928 04:50:44.450777 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1843,"thread_start_us":71,"threads_started":1}
I20260928 04:50:44.451233 13408 tablet_service.cc:1467] Processing CreateTablet for tablet 13b5e32f0b5c4eef99716815993b9527 (DEFAULT_TABLE table=test-table [id=681367205ac84e51a047707b80b38542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:50:44.451583 13408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b5e32f0b5c4eef99716815993b9527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:44.453691 13542 tablet_service.cc:1467] Processing CreateTablet for tablet 13b5e32f0b5c4eef99716815993b9527 (DEFAULT_TABLE table=test-table [id=681367205ac84e51a047707b80b38542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:50:44.454000 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b5e32f0b5c4eef99716815993b9527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:44.454986 13641 tablet_bootstrap.cc:492] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: Bootstrap starting.
I20260928 04:50:44.455552 13641 tablet_bootstrap.cc:654] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:44.455817 13641 log.cc:824] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:44.456487 13641 tablet_bootstrap.cc:492] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: No bootstrap required, opened a new log
I20260928 04:50:44.456540 13641 ts_tablet_manager.cc:1397] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:50:44.459069 13641 raft_consensus.cc:348] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 } } peers { permanent_uuid: "5184c234755943dfaed58a03c1f828e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34007 } } peers { permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 } }
I20260928 04:50:44.459233 13641 raft_consensus.cc:374] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:44.459266 13641 raft_consensus.cc:729] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97dd89cbf8a40d28fc288faa24e8c15, State: Initialized, Role: FOLLOWER
I20260928 04:50:44.459530 13641 consensus_queue.cc:260] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 } } peers { permanent_uuid: "5184c234755943dfaed58a03c1f828e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34007 } } peers { permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 } }
I20260928 04:50:44.459787 13641 ts_tablet_manager.cc:1428] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:50:44.459910 13474 heartbeater.cc:503] Master 127.0.98.62:34739 was elected leader, sending a full tablet report...
I20260928 04:50:44.460803 13643 tablet_bootstrap.cc:492] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: Bootstrap starting.
I20260928 04:50:44.461442 13643 tablet_bootstrap.cc:654] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:44.461740 13643 log.cc:824] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:44.461897 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.443791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34007 (local address 127.0.0.1:42020)
0928 04:50:44.445622 (+ 1831us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:44.445631 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:44.445679 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:44.457742 (+ 12063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:44.457743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:44.457746 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:44.457790 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:44.457792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.459113 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.459114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:44.460497 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:44.460500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.461624 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.461626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:44.461685 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.461692 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 04:50:44.461725 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 04:50:44.461767 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1800,"thread_start_us":60,"threads_started":1}
I20260928 04:50:44.462391 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.443902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.0.98.1:34007)
0928 04:50:44.457567 (+ 13665us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:44.457572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:44.457583 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:50:44.457618 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:44.457620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:44.457636 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:44.457858 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:44.457930 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.459010 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.460605 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.461459 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.462146 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.462259 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:44.462287 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 04:50:44.462323 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13629,"thread_start_us":60,"threads_started":1}
I20260928 04:50:44.463137 13255 tablet_service.cc:1467] Processing CreateTablet for tablet 13b5e32f0b5c4eef99716815993b9527 (DEFAULT_TABLE table=test-table [id=681367205ac84e51a047707b80b38542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:50:44.463410 13255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b5e32f0b5c4eef99716815993b9527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:44.465812 13643 tablet_bootstrap.cc:492] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: No bootstrap required, opened a new log
I20260928 04:50:44.465899 13643 ts_tablet_manager.cc:1397] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 04:50:44.468946 13643 raft_consensus.cc:348] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 } } peers { permanent_uuid: "5184c234755943dfaed58a03c1f828e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34007 } } peers { permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 } }
I20260928 04:50:44.469159 13643 raft_consensus.cc:374] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:44.469221 13643 raft_consensus.cc:729] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbaaf4736ae41579115c6e8f452e87f, State: Initialized, Role: FOLLOWER
I20260928 04:50:44.469362 13643 consensus_queue.cc:260] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 } } peers { permanent_uuid: "5184c234755943dfaed58a03c1f828e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34007 } } peers { permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 } }
I20260928 04:50:44.469641 13643 ts_tablet_manager.cc:1428] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260928 04:50:44.469753 13608 heartbeater.cc:503] Master 127.0.98.62:34739 was elected leader, sending a full tablet report...
I20260928 04:50:44.470132 13646 tablet_bootstrap.cc:492] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4: Bootstrap starting.
I20260928 04:50:44.470729 13646 tablet_bootstrap.cc:654] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:44.470990 13646 log.cc:824] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:44.471625 13646 tablet_bootstrap.cc:492] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4: No bootstrap required, opened a new log
I20260928 04:50:44.471689 13646 ts_tablet_manager.cc:1397] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:50:44.474109 13646 raft_consensus.cc:348] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 } } peers { permanent_uuid: "5184c234755943dfaed58a03c1f828e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34007 } } peers { permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 } }
I20260928 04:50:44.474279 13646 raft_consensus.cc:374] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:44.474309 13646 raft_consensus.cc:729] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5184c234755943dfaed58a03c1f828e4, State: Initialized, Role: FOLLOWER
I20260928 04:50:44.474480 13646 consensus_queue.cc:260] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 } } peers { permanent_uuid: "5184c234755943dfaed58a03c1f828e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34007 } } peers { permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 } }
I20260928 04:50:44.474697 13646 ts_tablet_manager.cc:1428] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:50:44.474782 13340 heartbeater.cc:503] Master 127.0.98.62:34739 was elected leader, sending a full tablet report...
W20260928 04:50:44.488180 13475 tablet.cc:2369] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 04:50:44 dist-test-slave-jccs krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 04:50:44.511396 13649 raft_consensus.cc:482] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:50:44.511523 13649 raft_consensus.cc:504] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 } } peers { permanent_uuid: "5184c234755943dfaed58a03c1f828e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34007 } } peers { permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 } }
I20260928 04:50:44.511924 13649 leader_election.cc:288] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfbaaf4736ae41579115c6e8f452e87f (127.0.98.3:33081), b97dd89cbf8a40d28fc288faa24e8c15 (127.0.98.2:34979)
I20260928 04:50:44.515826 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.512317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33585 (local address 127.0.98.2:34979)
0928 04:50:44.512354 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:44.512356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:44.512593 (+ 237us) server_negotiation.cc:407] Connection header received
0928 04:50:44.512675 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:44.512675 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:44.512687 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:44.512711 (+ 24us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:44.512895 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.513555 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.514984 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.515487 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.515530 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.515540 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:44.515725 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 04:50:44.515763 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:50:44.516219 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.512254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34979 (local address 127.0.98.1:33585)
0928 04:50:44.512549 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:44.512556 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:44.512612 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:44.512763 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:44.512764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:44.512765 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:44.512824 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:44.512825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.513678 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.513679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:44.514870 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:44.514872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.515593 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.515593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:44.515655 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.515658 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:50:44.516087 (+ 429us) client_negotiation.cc:240] Negotiation successful
0928 04:50:44.516127 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":51,"threads_started":1}
I20260928 04:50:44.516458 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.512124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35833 (local address 127.0.98.3:33081)
0928 04:50:44.512165 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:44.512168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:44.512490 (+ 322us) server_negotiation.cc:407] Connection header received
0928 04:50:44.512530 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:44.512531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:44.512544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:44.512603 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:44.512842 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.513594 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.515033 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.515599 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.515646 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.515656 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:44.516351 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 04:50:44.516391 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:50:44.516681 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.512037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33081 (local address 127.0.98.1:35833)
0928 04:50:44.512419 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:44.512429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:44.512484 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:44.512655 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:44.512656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:44.512658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:44.512747 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:44.512750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.513708 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.513709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:44.514926 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:44.514929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.516184 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.516186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:44.516253 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.516256 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:50:44.516566 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 04:50:44.516608 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":26,"thread_start_us":99,"threads_started":1}
I20260928 04:50:44.516630 13428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b5e32f0b5c4eef99716815993b9527" candidate_uuid: "5184c234755943dfaed58a03c1f828e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" is_pre_election: true
I20260928 04:50:44.516791 13428 raft_consensus.cc:2454] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5184c234755943dfaed58a03c1f828e4 in term 0.
I20260928 04:50:44.516952 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b5e32f0b5c4eef99716815993b9527" candidate_uuid: "5184c234755943dfaed58a03c1f828e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbaaf4736ae41579115c6e8f452e87f" is_pre_election: true
I20260928 04:50:44.517100 13562 raft_consensus.cc:2454] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5184c234755943dfaed58a03c1f828e4 in term 0.
I20260928 04:50:44.517331 13228 leader_election.cc:302] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5184c234755943dfaed58a03c1f828e4, b97dd89cbf8a40d28fc288faa24e8c15; no voters:
I20260928 04:50:44.517509 13649 raft_consensus.cc:2781] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:50:44.517550 13649 raft_consensus.cc:482] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:50:44.517571 13649 raft_consensus.cc:3037] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:44.518450 13649 raft_consensus.cc:504] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 } } peers { permanent_uuid: "5184c234755943dfaed58a03c1f828e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34007 } } peers { permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 } }
I20260928 04:50:44.518584 13649 leader_election.cc:288] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [CANDIDATE]: Term 1 election: Requested vote from peers dfbaaf4736ae41579115c6e8f452e87f (127.0.98.3:33081), b97dd89cbf8a40d28fc288faa24e8c15 (127.0.98.2:34979)
I20260928 04:50:44.518929 13428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b5e32f0b5c4eef99716815993b9527" candidate_uuid: "5184c234755943dfaed58a03c1f828e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b97dd89cbf8a40d28fc288faa24e8c15"
I20260928 04:50:44.519001 13428 raft_consensus.cc:3037] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:44.519001 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b5e32f0b5c4eef99716815993b9527" candidate_uuid: "5184c234755943dfaed58a03c1f828e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbaaf4736ae41579115c6e8f452e87f"
I20260928 04:50:44.519084 13562 raft_consensus.cc:3037] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:44.519912 13428 raft_consensus.cc:2454] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5184c234755943dfaed58a03c1f828e4 in term 1.
I20260928 04:50:44.519939 13562 raft_consensus.cc:2454] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5184c234755943dfaed58a03c1f828e4 in term 1.
I20260928 04:50:44.520150 13230 leader_election.cc:302] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5184c234755943dfaed58a03c1f828e4, dfbaaf4736ae41579115c6e8f452e87f; no voters:
I20260928 04:50:44.520232 13649 raft_consensus.cc:2781] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:50:44.520350 13649 raft_consensus.cc:686] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 1 LEADER]: Becoming Leader. State: Replica: 5184c234755943dfaed58a03c1f828e4, State: Running, Role: LEADER
I20260928 04:50:44.520440 13649 consensus_queue.cc:237] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 } } peers { permanent_uuid: "5184c234755943dfaed58a03c1f828e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34007 } } peers { permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 } }
I20260928 04:50:44.521245 13142 catalog_manager.cc:5495] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5184c234755943dfaed58a03c1f828e4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5184c234755943dfaed58a03c1f828e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5184c234755943dfaed58a03c1f828e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 } health_report { overall_health: UNKNOWN } } }
I20260928 04:50:44.532223 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.528454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.0.98.1:34007)
0928 04:50:44.529267 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:44.529269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:44.529278 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:50:44.529310 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:44.529310 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:44.529329 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:44.529398 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:50:44.529629 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.530233 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.531234 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:44.531507 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:44.531577 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:44.531738 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:50:44.531862 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:50:44.531938 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:44.532156 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 04:50:44.532174 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788}
I20260928 04:50:44.534070 13562 raft_consensus.cc:1261] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f [term 1 FOLLOWER]: Refusing update from remote peer 5184c234755943dfaed58a03c1f828e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:50:44.534081 13428 raft_consensus.cc:1261] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15 [term 1 FOLLOWER]: Refusing update from remote peer 5184c234755943dfaed58a03c1f828e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:50:44.534446 13649 consensus_queue.cc:1035] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfbaaf4736ae41579115c6e8f452e87f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:50:44.534621 13653 consensus_queue.cc:1035] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b97dd89cbf8a40d28fc288faa24e8c15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:50:44.538264 13658 mvcc.cc:205] Tried to move back new op lower bound from 7334178998409125888 to 7334178998356303872. Current Snapshot: MvccSnapshot[applied={T|T < 7334178998409125888}]
I20260928 04:50:44.538414 13657 mvcc.cc:205] Tried to move back new op lower bound from 7334178998409125888 to 7334178998356303872. Current Snapshot: MvccSnapshot[applied={T|T < 7334178998409125888}]
I20260928 04:50:44.540660 13141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:50:44.540766 13141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:50:44.541491 13141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28da0e9942a84239b05b6136125620af: Sending DeleteTablet for 3 replicas of tablet 13b5e32f0b5c4eef99716815993b9527
I20260928 04:50:44.541818 13542 tablet_service.cc:1514] Processing DeleteTablet for tablet 13b5e32f0b5c4eef99716815993b9527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:50:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38546
I20260928 04:50:44.541829 13255 tablet_service.cc:1514] Processing DeleteTablet for tablet 13b5e32f0b5c4eef99716815993b9527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:50:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42020
I20260928 04:50:44.542145 13408 tablet_service.cc:1514] Processing DeleteTablet for tablet 13b5e32f0b5c4eef99716815993b9527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:50:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47320
I20260928 04:50:44.542225 13666 tablet_replica.cc:330] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4: stopping tablet replica
I20260928 04:50:44.542320 13666 raft_consensus.cc:2229] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:50:44.542428 13666 raft_consensus.cc:2258] T 13b5e32f0b5c4eef99716815993b9527 P 5184c234755943dfaed58a03c1f828e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:50:44.542464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 13180
I20260928 04:50:44.545603 13668 tablet_replica.cc:330] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: stopping tablet replica
I20260928 04:50:44.545717 13668 raft_consensus.cc:2229] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:50:44.545856 13668 raft_consensus.cc:2258] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:50:44.546005 13667 tablet_replica.cc:330] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: stopping tablet replica
I20260928 04:50:44.546096 13667 raft_consensus.cc:2229] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:50:44.546190 13667 raft_consensus.cc:2258] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:50:44.546418 13668 ts_tablet_manager.cc:1905] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:50:44.546587 13667 ts_tablet_manager.cc:1905] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:50:44.548252 13667 ts_tablet_manager.cc:1918] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:50:44.548252 13668 ts_tablet_manager.cc:1918] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:50:44.548329 13667 log.cc:1196] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/13b5e32f0b5c4eef99716815993b9527
I20260928 04:50:44.548439 13668 log.cc:1196] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/13b5e32f0b5c4eef99716815993b9527
I20260928 04:50:44.548605 13667 ts_tablet_manager.cc:1939] T 13b5e32f0b5c4eef99716815993b9527 P dfbaaf4736ae41579115c6e8f452e87f: Deleting consensus metadata
I20260928 04:50:44.548820 13668 ts_tablet_manager.cc:1939] T 13b5e32f0b5c4eef99716815993b9527 P b97dd89cbf8a40d28fc288faa24e8c15: Deleting consensus metadata
I20260928 04:50:44.548990 13129 catalog_manager.cc:4841] TS dfbaaf4736ae41579115c6e8f452e87f (127.0.98.3:33081): tablet 13b5e32f0b5c4eef99716815993b9527 (table test-table [id=681367205ac84e51a047707b80b38542]) successfully deleted
I20260928 04:50:44.549310 13127 catalog_manager.cc:4841] TS b97dd89cbf8a40d28fc288faa24e8c15 (127.0.98.2:34979): tablet 13b5e32f0b5c4eef99716815993b9527 (table test-table [id=681367205ac84e51a047707b80b38542]) successfully deleted
W20260928 04:50:44.554170 13129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34007 (error 108)
I20260928 04:50:44.554937 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:44.554840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34007 (local address 127.0.0.1:42048)
0928 04:50:44.554882 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:44.554909 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 04:50:44.555037 13129 catalog_manager.cc:4584] TS 5184c234755943dfaed58a03c1f828e4 (127.0.98.1:34007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13b5e32f0b5c4eef99716815993b9527: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34007: connect: Connection refused (error 111)
I20260928 04:50:44.555383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 13344
I20260928 04:50:44.561350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 13477
I20260928 04:50:44.569368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 13109
I20260928 04:50:44.576162 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 04:50:44.930604 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 04:50:44.930719 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 04:50:45.275231 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 04:50:45.275 UTC [12176] LOG: received smart shutdown request
2026-09-28 04:50:45.279 UTC [12176] LOG: background worker "logical replication launcher" (PID 12185) exited with exit code 1
2026-09-28 04:50:45.279 UTC [12180] LOG: shutting down
2026-09-28 04:50:45.436 UTC [12176] LOG: database system is shut down
2026-09-28T04:50:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45645 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:50:45 dist-test-slave-jccs krb5kdc[13679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:50:45 dist-test-slave-jccs krb5kdc[13679](info): set up 2 sockets
Sep 28 04:50:45 dist-test-slave-jccs krb5kdc[13679](info): commencing operation
krb5kdc: starting...
W20260928 04:50:46.541312 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:50:46 dist-test-slave-jccs krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:50:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:50:46.654024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41489,127.0.98.61:43701,127.0.98.60:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:46.731377 13692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:46.731778 13692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:46.731817 13692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:46.733223 13692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:50:46.733285 13692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:46.733325 13692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:46.733354 13692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:50:46.733371 13692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:50:46.736236 13692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41489,127.0.98.61:43701,127.0.98.60:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:46.736480 13692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:46.736740 13692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:46.742997 13700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:46.743012 13698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:46.743012 13697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:46.743695 13692 server_base.cc:1034] running on GCE node
I20260928 04:50:46.743932 13692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:46.744472 13692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:46.745628 13692 hybrid_clock.cc:648] HybridClock initialized: now 1790571046745606 us; error 30 us; skew 500 ppm
Sep 28 04:50:46 dist-test-slave-jccs krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:46.748960 13692 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:50:46.749487 13692 webserver.cc:460] Webserver started at http://127.0.98.62:45395/ using document root <none> and password file <none>
I20260928 04:50:46.749743 13692 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:46.749789 13692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:46.749922 13692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:50:46.751233 13692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "ee612aec015b486882cdc0cc96770a1f"
format_stamp: "Formatted at 2026-09-28 04:50:46 on dist-test-slave-jccs"
server_key: "9f21045c2296011c13c6c40afdc6f675"
server_key_iv: "c0f25fc7908ec47cef3199de1af6cfa3"
server_key_version: "encryptionkey@0"
I20260928 04:50:46.751662 13692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee612aec015b486882cdc0cc96770a1f"
format_stamp: "Formatted at 2026-09-28 04:50:46 on dist-test-slave-jccs"
server_key: "9f21045c2296011c13c6c40afdc6f675"
server_key_iv: "c0f25fc7908ec47cef3199de1af6cfa3"
server_key_version: "encryptionkey@0"
I20260928 04:50:46.753485 13692 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 04:50:46.754289 13707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:46.754487 13692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:46.754582 13692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "ee612aec015b486882cdc0cc96770a1f"
format_stamp: "Formatted at 2026-09-28 04:50:46 on dist-test-slave-jccs"
server_key: "9f21045c2296011c13c6c40afdc6f675"
server_key_iv: "c0f25fc7908ec47cef3199de1af6cfa3"
server_key_version: "encryptionkey@0"
I20260928 04:50:46.754652 13692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:46.783893 13692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:46.785835 13692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:46.785996 13692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:46.790060 13692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41489
I20260928 04:50:46.790094 13759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41489 every 8 connection(s)
I20260928 04:50:46.790455 13692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:50:46.791240 13760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:46.792722 13760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:50:46.798569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 13692
Sep 28 04:50:46 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:46.798681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:50:46.798880 392 external_mini_cluster.cc:1428] Setting key b50b2e7608bc2b3639ecee20d7ecdc5f
I20260928 04:50:46.802263 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.793450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41489 (local address 127.0.0.1:42022)
0928 04:50:46.793633 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.793673 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:46.794132 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:46.794406 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:46.794409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:46.794666 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:46.794807 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:46.794811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.795788 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.795789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:46.796434 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:46.796436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.796943 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.796944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:46.797003 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:46.797536 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:46.797543 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:46.799071 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:46.800789 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:46.800795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:46.800798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:46.801063 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:46.801369 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:46.801370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:46.801370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:46.801447 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:46.801867 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:46.801871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:46.801930 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 04:50:46.801998 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:50:46.802111 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260928 04:50:46.802531 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.793455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.0.98.62:41489)
0928 04:50:46.793642 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:46.793651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:46.793778 (+ 127us) server_negotiation.cc:407] Connection header received
0928 04:50:46.794285 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:46.794287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:46.794304 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:46.794402 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:46.794900 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.795677 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.796531 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.796852 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.796992 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:46.799174 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:46.799181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:46.799181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:46.799187 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:46.800698 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:46.801172 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:46.801175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:46.801175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:46.801302 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:46.801511 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:46.801512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:46.801512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:46.801799 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:46.801885 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:46.802068 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 04:50:46.802198 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":145,"spinlock_wait_cycles":7552,"thread_start_us":96,"threads_started":1}
I20260928 04:50:46.803234 13760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:50:46.803560 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.803451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43701 (local address 127.0.0.1:35794)
0928 04:50:46.803497 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.803544 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:50:46.803673 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43701: connect: Connection refused (error 111)
I20260928 04:50:46.804060 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.803970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43701 (local address 127.0.0.1:35800)
0928 04:50:46.804017 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.804041 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:50:46.804162 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43701: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43701: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 04:50:46.822379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43701
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41489,127.0.98.61:43701,127.0.98.60:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:50:46.853355 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } attempt: 1
I20260928 04:50:46.853830 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.853699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43701 (local address 127.0.0.1:35802)
0928 04:50:46.853756 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.853800 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 04:50:46.854166 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.854073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43701 (local address 127.0.0.1:35818)
0928 04:50:46.854115 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.854140 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 04:50:46.854247 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43701: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43701: connect: Connection refused (error 111)
W20260928 04:50:46.900254 13767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:46.900483 13767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:46.900519 13767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:46.901747 13767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:50:46.901806 13767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:46.901829 13767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:46.901858 13767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:50:46.901889 13767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:50:46.903429 13767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41489,127.0.98.61:43701,127.0.98.60:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:46.903692 13767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:46.903913 13767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:46.908576 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:46.908588 13772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:46.908557 13775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:46.908939 13767 server_base.cc:1034] running on GCE node
I20260928 04:50:46.909226 13767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:46.909550 13767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:46.910712 13767 hybrid_clock.cc:648] HybridClock initialized: now 1790571046910696 us; error 42 us; skew 500 ppm
Sep 28 04:50:46 dist-test-slave-jccs krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:46.913343 13767 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 04:50:46.913805 13767 webserver.cc:460] Webserver started at http://127.0.98.61:40629/ using document root <none> and password file <none>
I20260928 04:50:46.914033 13767 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:46.914098 13767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:46.914237 13767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:50:46.915294 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/data/instance:
uuid: "0f40162307824524bf128c90d743dcf1"
format_stamp: "Formatted at 2026-09-28 04:50:46 on dist-test-slave-jccs"
server_key: "2774b87a2e57b078c639445283c6d1cc"
server_key_iv: "5c63eba538b85e220c2f76322a0ef71b"
server_key_version: "encryptionkey@0"
I20260928 04:50:46.915664 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/wal/instance:
uuid: "0f40162307824524bf128c90d743dcf1"
format_stamp: "Formatted at 2026-09-28 04:50:46 on dist-test-slave-jccs"
server_key: "2774b87a2e57b078c639445283c6d1cc"
server_key_iv: "5c63eba538b85e220c2f76322a0ef71b"
server_key_version: "encryptionkey@0"
I20260928 04:50:46.916942 13767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:50:46.917743 13782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:46.917917 13767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:46.918012 13767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/wal
uuid: "0f40162307824524bf128c90d743dcf1"
format_stamp: "Formatted at 2026-09-28 04:50:46 on dist-test-slave-jccs"
server_key: "2774b87a2e57b078c639445283c6d1cc"
server_key_iv: "5c63eba538b85e220c2f76322a0ef71b"
server_key_version: "encryptionkey@0"
I20260928 04:50:46.918099 13767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:46.922382 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } attempt: 2
I20260928 04:50:46.922741 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.922624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43701 (local address 127.0.0.1:35828)
0928 04:50:46.922678 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.922714 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:50:46.923026 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.922942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43701 (local address 127.0.0.1:35842)
0928 04:50:46.922984 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.923006 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 04:50:46.923110 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43701: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43701: connect: Connection refused (error 111)
I20260928 04:50:46.948725 13767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:46.950711 13767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:46.950872 13767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:46.954957 13767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43701
I20260928 04:50:46.955003 13834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43701 every 8 connection(s)
I20260928 04:50:46.955385 13767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
I20260928 04:50:46.956338 13835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:46.956815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 13767
I20260928 04:50:46.956923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-1/wal/instance
I20260928 04:50:46.957106 392 external_mini_cluster.cc:1428] Setting key 0d5e9250047d9a52ec136e78a9ecfbe6
I20260928 04:50:46.958715 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 04:50:46 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:46.970562 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.959524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.0.98.62:41489)
0928 04:50:46.959592 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:46.959595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:46.959829 (+ 234us) server_negotiation.cc:407] Connection header received
0928 04:50:46.960340 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:46.960341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:46.960356 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:46.960433 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:46.961098 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.962005 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.963001 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.963346 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.963427 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:46.967660 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:46.967661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:46.967662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:46.967670 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:46.969131 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:46.969612 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:46.969613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:46.969614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:46.969686 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:46.969915 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:46.969916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:46.969916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:46.970066 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:46.970138 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:46.970385 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 04:50:46.970468 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:50:46.970745 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.959513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41489 (local address 127.0.0.1:42028)
0928 04:50:46.959678 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.959720 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:46.960229 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:46.960516 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:46.960520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:46.960789 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:46.960987 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:46.960990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.962099 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.962099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:46.962905 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:46.962907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.963441 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.963442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:46.963544 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:46.964075 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:46.964092 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:46.967558 (+ 3466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:46.969290 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:46.969296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:46.969297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:46.969537 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:46.969756 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:46.969757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:46.969757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:46.969853 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:46.970131 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:46.970135 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:46.970245 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 04:50:46.970359 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 04:50:46.970511 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260928 04:50:46.971279 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:50:46 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:50:46.980340 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.971550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43701 (local address 127.0.0.1:35856)
0928 04:50:46.971606 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.971619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:46.971675 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:46.971877 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:46.971878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:46.972046 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:46.972110 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:46.972111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.972926 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.972927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:46.973863 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:46.973866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.974295 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.974296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:46.974375 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:46.974737 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:46.974741 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:46.977363 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:46.979265 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:46.979265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:46.979266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:46.979496 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:46.979679 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:46.979680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:46.979680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:46.979729 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:46.980096 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:46.980097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:46.980143 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 04:50:46.980198 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 04:50:46.980273 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:50:46.980511 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.971586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.0.98.61:43701)
0928 04:50:46.971735 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:46.971739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:46.971756 (+ 17us) server_negotiation.cc:407] Connection header received
0928 04:50:46.971785 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:46.971786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:46.971797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:46.971856 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:46.972185 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.972847 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.973946 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:46.974222 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:46.974280 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:46.977462 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:46.977470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:46.977471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:46.977477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:46.979175 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:46.979571 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:46.979574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:46.979575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:46.979628 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:46.979817 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:46.979818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:46.979818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:46.980028 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:46.980112 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:46.980224 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 04:50:46.980333 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260928 04:50:46.981093 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:50:46.981484 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.981339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52422)
0928 04:50:46.981399 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.981469 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:50:46.981604 13783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
I20260928 04:50:46.982059 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:46.981944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52428)
0928 04:50:46.981996 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:46.982043 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 04:50:46.982152 13835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38901: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 04:50:46.983704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38901
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41489,127.0.98.61:43701,127.0.98.60:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:50:47.014246 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } attempt: 3
Sep 28 04:50:47 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:50:47.021446 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.014656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43701 (local address 127.0.0.1:35860)
0928 04:50:47.014729 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.014744 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:47.014838 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:47.014989 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:47.014990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:47.015158 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:47.015220 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.015221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.015920 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.015920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.016428 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.016429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.016827 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.016828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.016914 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.017447 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:47.017451 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:47.018651 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:47.020321 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.020322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.020323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.020528 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.020730 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.020730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.020731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.020788 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.021089 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:47.021090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:47.021135 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 04:50:47.021245 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 04:50:47.021348 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 04:50:47.021446 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.014752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.0.98.61:43701)
0928 04:50:47.014820 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.014823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.014833 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:50:47.014916 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.014916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.014925 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.014984 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.015320 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.015827 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.016512 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.016717 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.016830 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.018749 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.018750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.018750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.018761 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.020221 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.020594 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.020595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.020595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.020657 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.020860 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.020861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.020861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.021003 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.021098 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.021271 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.021349 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:50:47.022068 13760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:50:47.022410 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.022316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52432)
0928 04:50:47.022362 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.022387 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 04:50:47.022670 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.022608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52438)
0928 04:50:47.022632 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.022652 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 04:50:47.022780 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38901: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
I20260928 04:50:47.031328 13835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } attempt: 1
I20260928 04:50:47.031680 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.031579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52452)
0928 04:50:47.031624 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.031651 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 04:50:47.032011 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.031890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52454)
0928 04:50:47.031921 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.031987 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 04:50:47.032510 13835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38901: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
I20260928 04:50:47.053915 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } attempt: 1
I20260928 04:50:47.054322 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.054206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52466)
0928 04:50:47.054256 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.054297 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 04:50:47.054626 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.054572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52482)
0928 04:50:47.054594 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.054614 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 04:50:47.054728 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38901: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
W20260928 04:50:47.069223 13842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:47.069420 13842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:47.069442 13842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:47.070520 13842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:50:47.070556 13842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:47.070571 13842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:47.070583 13842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:50:47.070595 13842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:50:47.072000 13842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41489,127.0.98.61:43701,127.0.98.60:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:47.072180 13842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:47.072341 13842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:47.077039 13848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:47.077034 13847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:47.077061 13850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:47.077039 13842 server_base.cc:1034] running on GCE node
I20260928 04:50:47.077472 13842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:47.077790 13842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:47.078958 13842 hybrid_clock.cc:648] HybridClock initialized: now 1790571047078942 us; error 42 us; skew 500 ppm
Sep 28 04:50:47 dist-test-slave-jccs krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:47.081391 13842 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 04:50:47.081801 13842 webserver.cc:460] Webserver started at http://127.0.98.60:46745/ using document root <none> and password file <none>
I20260928 04:50:47.082050 13842 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:47.082118 13842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:47.082267 13842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:50:47.083328 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/data/instance:
uuid: "b80fd1002e3f41af9d8484eaed05665a"
format_stamp: "Formatted at 2026-09-28 04:50:47 on dist-test-slave-jccs"
server_key: "a9ff5e949325a0066bb695ba0234aa41"
server_key_iv: "c6fc6c296dac38d04c65bba92749a493"
server_key_version: "encryptionkey@0"
I20260928 04:50:47.083671 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/wal/instance:
uuid: "b80fd1002e3f41af9d8484eaed05665a"
format_stamp: "Formatted at 2026-09-28 04:50:47 on dist-test-slave-jccs"
server_key: "a9ff5e949325a0066bb695ba0234aa41"
server_key_iv: "c6fc6c296dac38d04c65bba92749a493"
server_key_version: "encryptionkey@0"
I20260928 04:50:47.084944 13842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:50:47.085680 13857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:47.085836 13842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:47.085940 13842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/wal
uuid: "b80fd1002e3f41af9d8484eaed05665a"
format_stamp: "Formatted at 2026-09-28 04:50:47 on dist-test-slave-jccs"
server_key: "a9ff5e949325a0066bb695ba0234aa41"
server_key_iv: "c6fc6c296dac38d04c65bba92749a493"
server_key_version: "encryptionkey@0"
I20260928 04:50:47.086025 13842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:47.096282 13842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:47.098551 13842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:47.098702 13842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:47.100687 13835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } attempt: 2
I20260928 04:50:47.101157 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.100985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52492)
0928 04:50:47.101054 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.101119 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 04:50:47.101574 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.101450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52494)
0928 04:50:47.101478 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.101550 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 04:50:47.101675 13835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38901: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38901: connect: Connection refused (error 111)
I20260928 04:50:47.102685 13842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38901
I20260928 04:50:47.102738 13909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38901 every 8 connection(s)
I20260928 04:50:47.103032 13842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
I20260928 04:50:47.103818 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:47.105341 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:50:47.108078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 13842
I20260928 04:50:47.108160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790570897511889-392-0/minicluster-data/master-2/wal/instance
I20260928 04:50:47.108363 392 external_mini_cluster.cc:1428] Setting key 83d574beb90f8a2c419cbf90281e806b
Sep 28 04:50:47 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 04:50:47 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:50:47.114560 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.105998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.0.98.62:41489)
0928 04:50:47.106039 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.106041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.106243 (+ 202us) server_negotiation.cc:407] Connection header received
0928 04:50:47.106702 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.106703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.106714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.106760 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.107404 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.107874 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.109292 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.109471 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.109525 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.111677 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.111679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.111679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.111687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.113036 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.113512 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.113513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.113513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.113576 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.113811 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.113812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.113812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.113946 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.113998 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.114341 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.114421 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:50:47.114813 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.105960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41489 (local address 127.0.0.1:42042)
0928 04:50:47.106133 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.106163 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:47.106601 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:47.106821 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:47.106825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:47.107111 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:47.107296 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.107300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.107978 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.107979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.109183 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.109186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.109552 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.109553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.109720 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.110266 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:47.110274 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:47.111597 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:47.113118 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.113125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.113127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.113435 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.113661 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.113661 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.113661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.113748 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.114007 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:47.114012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:47.114181 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 04:50:47.114308 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 04:50:47.114485 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260928 04:50:47.115418 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:50:47.117527 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.108702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.0.98.60:38901)
0928 04:50:47.108891 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.108895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.108937 (+ 42us) server_negotiation.cc:407] Connection header received
0928 04:50:47.109009 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.109011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.109025 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.109082 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.109754 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.110587 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.111375 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.111715 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.111791 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.114340 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.114348 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.114349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.114354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.116347 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.116713 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.116715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.116716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.116778 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.116954 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.116955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.116955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.117102 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.117261 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.117333 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.117409 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
Sep 28 04:50:47 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:50:47.122422 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.115641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43701 (local address 127.0.0.1:35868)
0928 04:50:47.115683 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.115691 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:47.115728 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:47.115882 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:47.115883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:47.116011 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:47.116040 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.116041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.116713 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.116714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.117438 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.117440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.117958 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.117959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.118018 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.118434 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:47.118440 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:47.119719 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:47.121481 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.121482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.121482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.121641 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.121863 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.121864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.121864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.121917 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.122153 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:47.122154 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:47.122192 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 04:50:47.122257 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 04:50:47.122337 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 04:50:47.122429 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.115712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.0.98.61:43701)
0928 04:50:47.115763 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.115764 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.115785 (+ 21us) server_negotiation.cc:407] Connection header received
0928 04:50:47.115812 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.115813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.115821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.115858 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.116108 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.116644 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.117530 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.117839 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.117884 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.119802 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.119804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.119805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.119812 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.121397 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.121745 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.121746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.121746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.121800 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.121986 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.121988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.121988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.122088 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.122159 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.122283 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.122349 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 04:50:47 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:47.122709 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:50:47.125389 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.118491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.0.98.62:41489)
0928 04:50:47.118547 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.118549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.118556 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:50:47.118573 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.118574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.118582 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.118632 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.119235 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.119779 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.120615 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.120817 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.120867 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.123037 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.123038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.123038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.123043 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.124293 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.124656 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.124656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.124656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.124702 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.124868 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.124868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.124868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.124947 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.125137 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.125171 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.125288 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 04:50:47 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:50:47.129514 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.122930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52516)
0928 04:50:47.122979 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.122988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:47.123026 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:47.123277 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:47.123278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:47.123411 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:47.123443 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.123444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.124177 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.124178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.124930 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.124932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.125373 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.125374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.125435 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.125855 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:47.125860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:47.127003 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:47.128519 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.128520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.128520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.128685 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.128909 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.128910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.128911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.128967 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.129217 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:47.129218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:47.129258 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 04:50:47.129328 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:50:47.129413 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 04:50:47.129571 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.122951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.0.98.60:38901)
0928 04:50:47.122976 (+ 25us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.122978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.123196 (+ 218us) server_negotiation.cc:407] Connection header received
0928 04:50:47.123211 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.123212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.123220 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.123256 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.123503 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.124087 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.125006 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.125280 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.125372 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.127078 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.127079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.127079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.127085 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.128449 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.128787 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.128788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.128788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.128838 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.129029 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.129030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.129030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.129132 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.129237 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.129349 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.129462 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":13}
I20260928 04:50:47.130069 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } attempt: 2
Sep 28 04:50:47 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:50:47.131048 13910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a: Bootstrap starting.
I20260928 04:50:47.132184 13910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:47.132491 13910 log.cc:824] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:47.133448 13910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a: No bootstrap required, opened a new log
I20260928 04:50:47.134167 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.126289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.0.98.61:43701)
0928 04:50:47.126357 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.126359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.126367 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:50:47.126388 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.126389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.126397 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.126453 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.126908 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.127571 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.128333 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.128624 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.128706 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.131122 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.131123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.131123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.131129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.132815 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.133198 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.133199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.133200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.133252 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.133472 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.133473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.133473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.133652 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.133843 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.133882 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.133953 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 04:50:47 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:50:47.135792 13910 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } }
I20260928 04:50:47.135949 13910 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:47.136004 13910 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80fd1002e3f41af9d8484eaed05665a, State: Initialized, Role: FOLLOWER
I20260928 04:50:47.135767 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:50:47.136138 13910 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } }
I20260928 04:50:47.136416 13910 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:50:47.137892 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.130283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52532)
0928 04:50:47.130332 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.130341 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:47.130380 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:47.130503 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:47.130504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:47.130632 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:47.130660 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.130661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.131272 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.131273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.131907 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.131910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.133116 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.133117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.133241 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.133673 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:47.133678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:47.135020 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:47.136737 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.136738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.136739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.136935 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.137151 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.137152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.137152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.137233 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.137559 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:47.137560 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:47.137603 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 04:50:47.137669 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 04:50:47.137764 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 04:50:47.138375 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.130330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.0.98.60:38901)
0928 04:50:47.130360 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.130362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.130369 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:50:47.130438 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.130439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.130447 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.130500 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.130753 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.131194 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.131977 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.133032 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.133094 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.135105 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.135106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.135106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.135112 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.136643 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.137016 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.137017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.137017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.137076 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.137309 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.137309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.137310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.137471 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.137532 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.138208 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.138286 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:50:47.138957 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } } }
I20260928 04:50:47.139055 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:50:47.138712 13842 master_runner.cc:186] Error getting master registration for 127.0.98.62:41489: OK, instance_id { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" instance_seqno: 1790571046788908 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:50:47.141767 13842 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 04:50:47.138399 13934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:50:47.141965 13934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:50:47.142809 13760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f: Bootstrap starting.
I20260928 04:50:47.143514 13760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:47.143802 13760 log.cc:824] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:47.145080 13760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f: No bootstrap required, opened a new log
I20260928 04:50:47.146618 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.137592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.0.98.60:38901)
0928 04:50:47.138716 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.138719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.138726 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:50:47.138744 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.138744 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.138761 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.138818 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.139498 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.140342 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.141357 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.141678 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.141742 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.143236 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.143237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.143237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.143243 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.145219 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.145616 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.145617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.145618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.145689 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.145862 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.145863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.145863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.145983 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.146386 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.146424 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.146495 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1091,"thread_start_us":74,"threads_started":1}
I20260928 04:50:47.147652 13760 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } }
I20260928 04:50:47.147800 13760 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:47.147845 13760 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee612aec015b486882cdc0cc96770a1f, State: Initialized, Role: FOLLOWER
I20260928 04:50:47.147961 13760 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } }
I20260928 04:50:47.148252 13760 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:50:47.148451 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.137521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.0.98.62:41489)
0928 04:50:47.137558 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.137559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.138632 (+ 1073us) server_negotiation.cc:407] Connection header received
0928 04:50:47.138961 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.138962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.138972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.139000 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.139765 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.140536 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.141555 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.141821 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.141866 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.145454 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.145455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.145455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.145463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.147087 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.147465 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.147466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.147466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.147517 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.147744 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.147745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.147745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.147878 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.147963 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.148163 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.148233 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 04:50:47.150110 13949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:50:47.150209 13949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:50:47.150529 13938 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } } }
I20260928 04:50:47.150604 13938 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:50:47.151319 13692 master_runner.cc:186] Error getting master registration for 127.0.98.61:43701: OK, instance_id { permanent_uuid: "0f40162307824524bf128c90d743dcf1" instance_seqno: 1790571046953685 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:50:47.154137 13692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:50:47.155690 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.137827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.0.98.61:43701)
0928 04:50:47.137865 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.137867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.138590 (+ 723us) server_negotiation.cc:407] Connection header received
0928 04:50:47.138661 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.138662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.138672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.138798 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.140167 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.140842 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.149290 (+ 8448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.149584 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.149666 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.151076 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.151078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.151078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.151088 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.154423 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.154840 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.154841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.154841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.154922 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.155141 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.155142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.155142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.155273 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.155365 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.155517 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.155596 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:50:47.192834 13835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } attempt: 3
Sep 28 04:50:47 dist-test-slave-jccs krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:50:47.201246 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.193125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38901 (local address 127.0.0.1:52540)
0928 04:50:47.193218 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:47.193230 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:47.193268 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:47.193439 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:47.193440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:47.193636 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:47.193688 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.193690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.194676 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.194677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.195476 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:47.195479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.196057 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.196058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:47.196117 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.196728 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:47.196734 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:47.198285 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:47.200101 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.200102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.200103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.200333 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.200578 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:47.200579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:47.200579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:47.200641 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:47.200913 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:47.200914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:47.200956 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 04:50:47.201022 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 04:50:47.201124 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 04:50:47.201273 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:47.193218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.0.98.60:38901)
0928 04:50:47.193260 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:47.193262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:47.193271 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:50:47.193348 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:47.193348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:47.193359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:47.193432 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:47.193790 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.194580 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.195578 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:47.195925 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:47.195996 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:47.198384 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:47.198386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:47.198386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:47.198394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:47.200018 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.200444 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.200445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.200446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.200509 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:47.200705 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:47.200706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:47.200706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:47.200847 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:47.200913 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:47.201064 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 04:50:47.201156 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:50:47.203090 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1: Bootstrap starting.
I20260928 04:50:47.203797 13835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:47.204108 13835 log.cc:824] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:47.204969 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1: No bootstrap required, opened a new log
I20260928 04:50:47.206909 13835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } }
I20260928 04:50:47.207063 13835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:47.207095 13835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f40162307824524bf128c90d743dcf1, State: Initialized, Role: FOLLOWER
I20260928 04:50:47.207233 13835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } }
I20260928 04:50:47.207516 13835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:50:47.207499 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } } }
I20260928 04:50:47.207726 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:50:47.209244 13963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:50:47.209355 13963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:50:47.211086 13767 master_runner.cc:263] No leader master found from master 0f40162307824524bf128c90d743dcf1
I20260928 04:50:47.211138 13767 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 04:50:47.286800 13713 tablet.cc:2369] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:50:47.331672 13938 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:50:47.331807 13938 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } }
I20260928 04:50:47.332056 13938 leader_election.cc:288] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f40162307824524bf128c90d743dcf1 (127.0.98.61:43701), b80fd1002e3f41af9d8484eaed05665a (127.0.98.60:38901)
I20260928 04:50:47.332280 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee612aec015b486882cdc0cc96770a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80fd1002e3f41af9d8484eaed05665a" is_pre_election: true
I20260928 04:50:47.332340 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee612aec015b486882cdc0cc96770a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f40162307824524bf128c90d743dcf1" is_pre_election: true
I20260928 04:50:47.332444 13884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee612aec015b486882cdc0cc96770a1f in term 0.
I20260928 04:50:47.332456 13809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee612aec015b486882cdc0cc96770a1f in term 0.
I20260928 04:50:47.332618 13708 leader_election.cc:302] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b80fd1002e3f41af9d8484eaed05665a, ee612aec015b486882cdc0cc96770a1f; no voters:
I20260928 04:50:47.332747 13938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:50:47.332798 13938 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:50:47.332820 13938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:47.333616 13938 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } }
I20260928 04:50:47.333750 13938 leader_election.cc:288] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [CANDIDATE]: Term 1 election: Requested vote from peers 0f40162307824524bf128c90d743dcf1 (127.0.98.61:43701), b80fd1002e3f41af9d8484eaed05665a (127.0.98.60:38901)
I20260928 04:50:47.333899 13809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee612aec015b486882cdc0cc96770a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f40162307824524bf128c90d743dcf1"
I20260928 04:50:47.333935 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee612aec015b486882cdc0cc96770a1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80fd1002e3f41af9d8484eaed05665a"
I20260928 04:50:47.333992 13809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:47.334035 13884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:47.334651 13809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee612aec015b486882cdc0cc96770a1f in term 1.
I20260928 04:50:47.334803 13710 leader_election.cc:302] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f40162307824524bf128c90d743dcf1, ee612aec015b486882cdc0cc96770a1f; no voters:
I20260928 04:50:47.334897 13938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:50:47.334954 13884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee612aec015b486882cdc0cc96770a1f in term 1.
I20260928 04:50:47.335044 13938 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [term 1 LEADER]: Becoming Leader. State: Replica: ee612aec015b486882cdc0cc96770a1f, State: Running, Role: LEADER
I20260928 04:50:47.335129 13938 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } }
I20260928 04:50:47.335510 13938 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee612aec015b486882cdc0cc96770a1f. Latest consensus state: current_term: 1 leader_uuid: "ee612aec015b486882cdc0cc96770a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } } }
I20260928 04:50:47.335579 13938 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:47.335934 13967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:50:47.336112 13967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:50:47.337489 13884 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [term 1 FOLLOWER]: Refusing update from remote peer ee612aec015b486882cdc0cc96770a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:50:47.337532 13809 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [term 1 FOLLOWER]: Refusing update from remote peer ee612aec015b486882cdc0cc96770a1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:50:47.337877 13938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:50:47.338006 13965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:50:47.339463 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee612aec015b486882cdc0cc96770a1f. Latest consensus state: current_term: 1 leader_uuid: "ee612aec015b486882cdc0cc96770a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } } }
I20260928 04:50:47.339569 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:50:47.339602 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee612aec015b486882cdc0cc96770a1f. Latest consensus state: current_term: 1 leader_uuid: "ee612aec015b486882cdc0cc96770a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } } }
I20260928 04:50:47.339665 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:50:47.340773 13938 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee612aec015b486882cdc0cc96770a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } } }
I20260928 04:50:47.340857 13938 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:47.340940 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee612aec015b486882cdc0cc96770a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } } }
I20260928 04:50:47.341023 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:50:47.341369 13967 catalog_manager.cc:1347] Generated new cluster ID: b48a83486f2347169544e52c1cfd4815
I20260928 04:50:47.341418 13967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:50:47.341404 13938 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee612aec015b486882cdc0cc96770a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } } }
I20260928 04:50:47.341486 13938 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:47.341629 13969 mvcc.cc:205] Tried to move back new op lower bound from 7334179009892306944 to 7334179009885339648. Current Snapshot: MvccSnapshot[applied={T|T < 7334179009892306944 or (T in {7334179009892306944})}]
I20260928 04:50:47.341847 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee612aec015b486882cdc0cc96770a1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee612aec015b486882cdc0cc96770a1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41489 } } peers { permanent_uuid: "0f40162307824524bf128c90d743dcf1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43701 } } peers { permanent_uuid: "b80fd1002e3f41af9d8484eaed05665a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38901 } } }
I20260928 04:50:47.341912 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:50:47.342133 13970 mvcc.cc:205] Tried to move back new op lower bound from 7334179009892306944 to 7334179009885339648. Current Snapshot: MvccSnapshot[applied={T|T < 7334179009892306944 or (T in {7334179009892306944})}]
I20260928 04:50:47.358302 13967 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:50:47.359045 13967 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:50:47.367115 13967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee612aec015b486882cdc0cc96770a1f: Generated new TSK 0
I20260928 04:50:47.367332 13967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:50:48.142385 13934 catalog_manager.cc:1259] Loaded cluster ID: b48a83486f2347169544e52c1cfd4815
I20260928 04:50:48.142448 13934 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a: loading cluster ID for follower catalog manager: success
I20260928 04:50:48.143283 13934 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a: acquiring CA information for follower catalog manager: success
I20260928 04:50:48.143587 13934 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b80fd1002e3f41af9d8484eaed05665a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:50:48.209836 13963 catalog_manager.cc:1259] Loaded cluster ID: b48a83486f2347169544e52c1cfd4815
I20260928 04:50:48.209910 13963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1: loading cluster ID for follower catalog manager: success
I20260928 04:50:48.210649 13963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1: acquiring CA information for follower catalog manager: success
I20260928 04:50:48.210887 13963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f40162307824524bf128c90d743dcf1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:50:48.431143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 13692
I20260928 04:50:48.435850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 13767
I20260928 04:50:48.440655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 13842
2026-09-28T04:50:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2904 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:50:48 dist-test-slave-jccs krb5kdc[13982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:50:48 dist-test-slave-jccs krb5kdc[13982](info): set up 2 sockets
Sep 28 04:50:48 dist-test-slave-jccs krb5kdc[13982](info): commencing operation
krb5kdc: starting...
W20260928 04:50:49.441386 392 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:50:49 dist-test-slave-jccs krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:50:49.550222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:49.628047 13995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:49.628293 13995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:49.628328 13995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:49.629478 13995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:50:49.629544 13995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:49.629567 13995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:49.629597 13995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:50:49.629639 13995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:50:49.631022 13995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:49.631296 13995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:49.631515 13995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:49.636138 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:49.636193 13995 server_base.cc:1034] running on GCE node
W20260928 04:50:49.636124 14003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:49.636117 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:49.636494 13995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:49.636792 13995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:49.637934 13995 hybrid_clock.cc:648] HybridClock initialized: now 1790571049637923 us; error 27 us; skew 500 ppm
Sep 28 04:50:49 dist-test-slave-jccs krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:49.640359 13995 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:50:49.640751 13995 webserver.cc:460] Webserver started at http://127.0.98.62:36311/ using document root <none> and password file <none>
I20260928 04:50:49.640964 13995 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:49.641026 13995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:49.641206 13995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:50:49.642216 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "4d3139b66ca34ef6a1fe12312aec6ddb"
format_stamp: "Formatted at 2026-09-28 04:50:49 on dist-test-slave-jccs"
server_key: "65a2922896057bbc04ce57de95a33295"
server_key_iv: "1ed31741d232a36fd7e2c4a5b1523a2b"
server_key_version: "encryptionkey@0"
I20260928 04:50:49.642563 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d3139b66ca34ef6a1fe12312aec6ddb"
format_stamp: "Formatted at 2026-09-28 04:50:49 on dist-test-slave-jccs"
server_key: "65a2922896057bbc04ce57de95a33295"
server_key_iv: "1ed31741d232a36fd7e2c4a5b1523a2b"
server_key_version: "encryptionkey@0"
I20260928 04:50:49.643800 13995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:50:49.644505 14010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:49.644690 13995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:50:49.644793 13995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "4d3139b66ca34ef6a1fe12312aec6ddb"
format_stamp: "Formatted at 2026-09-28 04:50:49 on dist-test-slave-jccs"
server_key: "65a2922896057bbc04ce57de95a33295"
server_key_iv: "1ed31741d232a36fd7e2c4a5b1523a2b"
server_key_version: "encryptionkey@0"
I20260928 04:50:49.644873 13995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:49.654482 13995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:49.656551 13995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:49.656698 13995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:49.660429 13995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35815
I20260928 04:50:49.660458 14062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35815 every 8 connection(s)
I20260928 04:50:49.660776 13995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:50:49.661708 14063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:49.663808 14063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb: Bootstrap starting.
I20260928 04:50:49.664369 14063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:49.664527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 13995
I20260928 04:50:49.664638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:50:49.664710 14063 log.cc:824] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:49.664844 392 external_mini_cluster.cc:1428] Setting key 4f88b802bc2f51962ee47df4bf8918bf
I20260928 04:50:49.665500 14063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb: No bootstrap required, opened a new log
I20260928 04:50:49.667383 14063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3139b66ca34ef6a1fe12312aec6ddb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } }
I20260928 04:50:49.667529 14063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:49.667572 14063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3139b66ca34ef6a1fe12312aec6ddb, State: Initialized, Role: FOLLOWER
I20260928 04:50:49.667685 14063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3139b66ca34ef6a1fe12312aec6ddb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } }
I20260928 04:50:49.667778 14063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:50:49.667819 14063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:50:49.667865 14063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:49.668397 14063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3139b66ca34ef6a1fe12312aec6ddb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } }
I20260928 04:50:49.668510 14063 leader_election.cc:302] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3139b66ca34ef6a1fe12312aec6ddb; no voters:
I20260928 04:50:49.668659 14063 leader_election.cc:288] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:50:49.668701 14068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:50:49.668844 14068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [term 1 LEADER]: Becoming Leader. State: Replica: 4d3139b66ca34ef6a1fe12312aec6ddb, State: Running, Role: LEADER
I20260928 04:50:49.668908 14063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:50:49.668926 14068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3139b66ca34ef6a1fe12312aec6ddb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } }
I20260928 04:50:49.669241 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3139b66ca34ef6a1fe12312aec6ddb. Latest consensus state: current_term: 1 leader_uuid: "4d3139b66ca34ef6a1fe12312aec6ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3139b66ca34ef6a1fe12312aec6ddb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } } }
I20260928 04:50:49.669344 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:49.669245 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d3139b66ca34ef6a1fe12312aec6ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3139b66ca34ef6a1fe12312aec6ddb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35815 } } }
I20260928 04:50:49.669492 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:49.669549 14074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:50:49.670012 14074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:50:49.671265 14085 mvcc.cc:205] Tried to move back new op lower bound from 7334179019451793408 to 7334179019444248576. Current Snapshot: MvccSnapshot[applied={T|T < 7334179019451793408}]
Sep 28 04:50:49 dist-test-slave-jccs krb5kdc[13982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:49.672741 14074 catalog_manager.cc:1347] Generated new cluster ID: c15638df7d194377adc96535af8e6cd1
I20260928 04:50:49.672791 14074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:50:49.676110 14067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:49.665285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.0.98.62:35815)
0928 04:50:49.665461 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:49.665469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:49.665498 (+ 29us) server_negotiation.cc:407] Connection header received
0928 04:50:49.665932 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:49.665938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:49.665979 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:49.666109 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:49.666680 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:49.667495 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:49.668381 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:49.668988 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:49.669421 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:49.672611 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:49.672615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:49.672615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:49.672626 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:49.674543 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:49.674865 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:49.674868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:49.674869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:49.674949 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:49.675104 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:49.675105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:49.675105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:49.675342 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:49.675462 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:49.675818 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 04:50:49.675929 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260928 04:50:49.677024 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:50:49.694053 14074 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:50:49.694595 14074 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:50:49.697813 14074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d3139b66ca34ef6a1fe12312aec6ddb: Generated new TSK 0
I20260928 04:50:49.697988 14074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:50:49.730846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 13995
2026-09-28T04:50:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1288 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:50:49 dist-test-slave-jccs krb5kdc[14090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:50:49 dist-test-slave-jccs krb5kdc[14090](info): set up 2 sockets
Sep 28 04:50:49 dist-test-slave-jccs krb5kdc[14090](info): commencing operation
krb5kdc: starting...
W20260928 04:50:50.728968 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:50:50 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:50:50.840519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40305
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:50.915243 14103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:50.915472 14103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:50.915503 14103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:50.916507 14103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:50:50.916571 14103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:50.916591 14103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:50.916610 14103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:50:50.916637 14103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:50:50.917923 14103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:50.918162 14103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:50.918362 14103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:50.922991 14111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:50.923048 14103 server_base.cc:1034] running on GCE node
W20260928 04:50:50.923007 14109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:50.922997 14108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:50.923473 14103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:50.923738 14103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:50.924875 14103 hybrid_clock.cc:648] HybridClock initialized: now 1790571050924862 us; error 28 us; skew 500 ppm
Sep 28 04:50:50 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:50.927238 14103 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:50:50.927632 14103 webserver.cc:460] Webserver started at http://127.0.98.62:41323/ using document root <none> and password file <none>
I20260928 04:50:50.927891 14103 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:50.927935 14103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:50.928068 14103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:50:50.929060 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "6dcb8e1046d94b768c5713bf11e18f27"
format_stamp: "Formatted at 2026-09-28 04:50:50 on dist-test-slave-jccs"
server_key: "f58d2f938d63880f7add844bf355e95a"
server_key_iv: "a12c9f7dcfb3993bf19d3f6e9a20358b"
server_key_version: "encryptionkey@0"
I20260928 04:50:50.929450 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "6dcb8e1046d94b768c5713bf11e18f27"
format_stamp: "Formatted at 2026-09-28 04:50:50 on dist-test-slave-jccs"
server_key: "f58d2f938d63880f7add844bf355e95a"
server_key_iv: "a12c9f7dcfb3993bf19d3f6e9a20358b"
server_key_version: "encryptionkey@0"
I20260928 04:50:50.930660 14103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:50.931264 14118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:50.931452 14103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:50:50.931545 14103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "6dcb8e1046d94b768c5713bf11e18f27"
format_stamp: "Formatted at 2026-09-28 04:50:50 on dist-test-slave-jccs"
server_key: "f58d2f938d63880f7add844bf355e95a"
server_key_iv: "a12c9f7dcfb3993bf19d3f6e9a20358b"
server_key_version: "encryptionkey@0"
I20260928 04:50:50.931627 14103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:50.948344 14103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:50.950090 14103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:50.950237 14103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:50.953701 14103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40305
I20260928 04:50:50.953723 14170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40305 every 8 connection(s)
I20260928 04:50:50.954066 14103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:50:50.954542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 14103
I20260928 04:50:50.954682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:50:50.954954 392 external_mini_cluster.cc:1428] Setting key dfa705b9a749a22550f7ae61d97fc370
I20260928 04:50:50.955199 14171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:50.958412 14171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Bootstrap starting.
I20260928 04:50:50.959041 14171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:50.959357 14171 log.cc:824] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:50.960157 14171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: No bootstrap required, opened a new log
Sep 28 04:50:50 dist-test-slave-jccs krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:50.962204 14171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:50.962335 14171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:50.962404 14171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcb8e1046d94b768c5713bf11e18f27, State: Initialized, Role: FOLLOWER
I20260928 04:50:50.962535 14171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:50.962618 14171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:50:50.962653 14171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:50:50.962707 14171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:50.963200 14171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:50.963335 14171 leader_election.cc:302] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dcb8e1046d94b768c5713bf11e18f27; no voters:
I20260928 04:50:50.963515 14171 leader_election.cc:288] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:50:50.963542 14176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:50:50.963651 14176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 1 LEADER]: Becoming Leader. State: Replica: 6dcb8e1046d94b768c5713bf11e18f27, State: Running, Role: LEADER
I20260928 04:50:50.963768 14171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:50:50.963789 14176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:50.964187 14176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dcb8e1046d94b768c5713bf11e18f27. Latest consensus state: current_term: 1 leader_uuid: "6dcb8e1046d94b768c5713bf11e18f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } } }
I20260928 04:50:50.964198 14177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dcb8e1046d94b768c5713bf11e18f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } } }
I20260928 04:50:50.964283 14176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:50.964380 14177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:50.964603 14185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:50:50.965160 14185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:50:50.966076 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:50.955456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.0.98.62:40305)
0928 04:50:50.955619 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:50.955626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:50.955650 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:50:50.956046 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:50.956052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:50.956091 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:50.956206 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:50.956905 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:50.957758 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:50.958856 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:50.959277 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:50.959354 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:50.961904 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:50.961908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:50.961908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:50.961917 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:50.963815 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:50.964178 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:50.964181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:50.964181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:50.964343 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:50.964781 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:50.964781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:50.964782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:50.965096 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:50.965697 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:50.965764 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 04:50:50.965896 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260928 04:50:50.967056 14185 catalog_manager.cc:1347] Generated new cluster ID: 83a7b8c4122f4e0f9d1b1a07f4620c70
I20260928 04:50:50.967103 14185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:50:50.984227 14185 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:50:50.984890 14185 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:50:50.990715 14185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Generated new TSK 0
I20260928 04:50:50.990895 14185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:50:51.018078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40305
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:51.097415 14197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:51.097620 14197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:51.097656 14197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:51.098723 14197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:51.098785 14197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:51.098830 14197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:50:51.100212 14197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:51.100474 14197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:51.100685 14197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:51.105288 14202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:51.105327 14205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:51.105432 14197 server_base.cc:1034] running on GCE node
W20260928 04:50:51.105300 14203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:51.105706 14197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:51.105984 14197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:51.107136 14197 hybrid_clock.cc:648] HybridClock initialized: now 1790571051107135 us; error 36 us; skew 500 ppm
Sep 28 04:50:51 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:51.109411 14197 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:50:51.109827 14197 webserver.cc:460] Webserver started at http://127.0.98.1:38331/ using document root <none> and password file <none>
I20260928 04:50:51.110021 14197 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:51.110080 14197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:51.110224 14197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:50:51.111132 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "317807aa99a7476d954a352fcf1a075e"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "c00963a0ae8d63d570e0224beaa54359"
server_key_iv: "945e1ce0a2f5fdbbeea7dd27e47d3c30"
server_key_version: "encryptionkey@0"
I20260928 04:50:51.111454 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "317807aa99a7476d954a352fcf1a075e"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "c00963a0ae8d63d570e0224beaa54359"
server_key_iv: "945e1ce0a2f5fdbbeea7dd27e47d3c30"
server_key_version: "encryptionkey@0"
I20260928 04:50:51.112736 14197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:50:51.113425 14212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:51.113592 14197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:50:51.113683 14197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "317807aa99a7476d954a352fcf1a075e"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "c00963a0ae8d63d570e0224beaa54359"
server_key_iv: "945e1ce0a2f5fdbbeea7dd27e47d3c30"
server_key_version: "encryptionkey@0"
I20260928 04:50:51.113762 14197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:51.125288 14197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:51.126919 14197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:51.127068 14197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:51.127287 14197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:51.127566 14197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:51.127614 14197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:51.127650 14197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:51.127674 14197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:51.133164 14197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36883
I20260928 04:50:51.133289 14325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36883 every 8 connection(s)
I20260928 04:50:51.133535 14197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:50:51 dist-test-slave-jccs krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:51.141593 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.134411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51379 (local address 127.0.98.62:40305)
0928 04:50:51.134452 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:51.134454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:51.134463 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:50:51.134855 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:51.134856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:51.134865 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:51.134917 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:51.135790 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.136248 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.137062 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.137254 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.137298 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.139227 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:51.139228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:51.139228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:51.139233 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:51.140450 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:51.140804 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:51.140805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:51.140805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:51.140858 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:51.141063 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:51.141064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:51.141064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:51.141153 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:51.141261 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:51.141466 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 04:50:51.141531 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:50:51.141762 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.134192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:51379)
0928 04:50:51.134322 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:51.134355 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:51.134770 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:51.134964 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:51.134967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:51.135545 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:51.135715 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.135720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.136319 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.136320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.137000 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.137001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.137312 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.137313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.137403 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.137879 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:51.137886 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:51.139154 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:51.140536 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:51.140541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:51.140543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:51.140743 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:51.140925 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:51.140925 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:51.140926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:51.141013 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:51.141257 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:51.141261 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:51.141362 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 04:50:51.141441 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 04:50:51.141546 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260928 04:50:51.142181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 14197
I20260928 04:50:51.142249 14326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40305
I20260928 04:50:51.142304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:50:51.142339 14326 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:51.142505 392 external_mini_cluster.cc:1428] Setting key ea23498a84a749ff5aca0861c08f6973
I20260928 04:50:51.142503 14326 heartbeater.cc:511] Master 127.0.98.62:40305 requested a full tablet report, sending...
I20260928 04:50:51.143056 14134 ts_manager.cc:194] Registered new tserver with Master: 317807aa99a7476d954a352fcf1a075e (127.0.98.1:36883)
I20260928 04:50:51.143616 14134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51379
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:50:51.165158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40305
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:51.240525 14331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:51.240688 14331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:51.240710 14331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:51.241796 14331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:51.241830 14331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:51.241854 14331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:50:51.243096 14331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:51.243244 14331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:51.243393 14331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:51.247864 14336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:51.247892 14337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:51.247870 14339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:51.248030 14331 server_base.cc:1034] running on GCE node
I20260928 04:50:51.248273 14331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:51.248550 14331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:51.249701 14331 hybrid_clock.cc:648] HybridClock initialized: now 1790571051249697 us; error 38 us; skew 500 ppm
Sep 28 04:50:51 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:51.251945 14331 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:50:51.252382 14331 webserver.cc:460] Webserver started at http://127.0.98.2:34837/ using document root <none> and password file <none>
I20260928 04:50:51.252597 14331 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:51.252638 14331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:51.252771 14331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:50:51.253839 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "170cf5c796ff4af9a9e295aab2087cc3"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "ba4058e75d09567d3af0015086aae8ca"
server_key_iv: "32ac7d0ec544c600d19c4d971510ae0a"
server_key_version: "encryptionkey@0"
I20260928 04:50:51.254168 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "170cf5c796ff4af9a9e295aab2087cc3"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "ba4058e75d09567d3af0015086aae8ca"
server_key_iv: "32ac7d0ec544c600d19c4d971510ae0a"
server_key_version: "encryptionkey@0"
I20260928 04:50:51.255357 14331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:50:51.255947 14346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:51.256120 14331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:50:51.256203 14331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "170cf5c796ff4af9a9e295aab2087cc3"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "ba4058e75d09567d3af0015086aae8ca"
server_key_iv: "32ac7d0ec544c600d19c4d971510ae0a"
server_key_version: "encryptionkey@0"
I20260928 04:50:51.256268 14331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:51.267927 14331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:51.269667 14331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:51.269805 14331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:51.270023 14331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:51.270318 14331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:51.270375 14331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:51.270408 14331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:51.270434 14331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:51.275967 14331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44437
I20260928 04:50:51.276013 14459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44437 every 8 connection(s)
I20260928 04:50:51.276270 14331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:50:51.279321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 14331
I20260928 04:50:51.279407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:50:51.279609 392 external_mini_cluster.cc:1428] Setting key 906a72cd77237c5710da2b7aac80c2e0
Sep 28 04:50:51 dist-test-slave-jccs krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:51.285974 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.276862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33261 (local address 127.0.98.62:40305)
0928 04:50:51.276934 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:51.276937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:51.277135 (+ 198us) server_negotiation.cc:407] Connection header received
0928 04:50:51.277642 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:51.277643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:51.277662 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:51.277710 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:51.278555 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.279082 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.280444 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.280615 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.280718 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.283101 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:51.283103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:51.283103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:51.283109 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:51.284527 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:51.285019 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:51.285020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:51.285020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:51.285073 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:51.285371 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:51.285371 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:51.285372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:51.285461 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:51.285508 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:51.285847 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 04:50:51.285906 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:50:51.286199 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.276814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:33261)
0928 04:50:51.277016 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:51.277049 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:51.277544 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:51.277771 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:51.277779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:51.278285 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:51.278442 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.278445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.279210 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.279211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.280365 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.280368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.280725 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.280726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.280849 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.281340 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:51.281350 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:51.283007 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:51.284610 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:51.284617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:51.284619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:51.284946 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:51.285149 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:51.285150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:51.285151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:51.285303 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:51.285525 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:51.285531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:51.285696 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 04:50:51.285800 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:50:51.285992 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":107,"threads_started":1}
I20260928 04:50:51.286638 14460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40305
I20260928 04:50:51.286723 14460 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:51.286899 14460 heartbeater.cc:511] Master 127.0.98.62:40305 requested a full tablet report, sending...
I20260928 04:50:51.287214 14135 ts_manager.cc:194] Registered new tserver with Master: 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437)
I20260928 04:50:51.287552 14135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33261
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:50:51.303617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40305
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:51.379635 14465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:51.379894 14465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:51.379928 14465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:51.381006 14465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:51.381071 14465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:51.381115 14465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:50:51.382534 14465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:51.382774 14465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:51.382992 14465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:50:51.387573 14465 server_base.cc:1034] running on GCE node
W20260928 04:50:51.387547 14471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:51.387547 14473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:51.387542 14470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:51.387867 14465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:51.388151 14465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:51.389283 14465 hybrid_clock.cc:648] HybridClock initialized: now 1790571051389266 us; error 29 us; skew 500 ppm
Sep 28 04:50:51 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:51.391543 14465 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:50:51.391894 14465 webserver.cc:460] Webserver started at http://127.0.98.3:41463/ using document root <none> and password file <none>
I20260928 04:50:51.392051 14465 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:51.392087 14465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:51.392167 14465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:50:51.393038 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "f405870892a3503356aa30568ed8068c"
server_key_iv: "46dc594d2eb06ca92f66063b59328112"
server_key_version: "encryptionkey@0"
I20260928 04:50:51.393365 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "f405870892a3503356aa30568ed8068c"
server_key_iv: "46dc594d2eb06ca92f66063b59328112"
server_key_version: "encryptionkey@0"
I20260928 04:50:51.394465 14465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:51.395117 14480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:51.395287 14465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:50:51.395375 14465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "f405870892a3503356aa30568ed8068c"
server_key_iv: "46dc594d2eb06ca92f66063b59328112"
server_key_version: "encryptionkey@0"
I20260928 04:50:51.395428 14465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:51.407260 14465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:51.409585 14465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:51.409703 14465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:51.409894 14465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:51.410310 14465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:51.410375 14465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:51.410418 14465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:51.410465 14465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:51.415757 14465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43703
I20260928 04:50:51.415829 14593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43703 every 8 connection(s)
I20260928 04:50:51.416095 14465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:50:51.417732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 14465
I20260928 04:50:51.417805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:50:51.417968 392 external_mini_cluster.cc:1428] Setting key de2fad22b8897a197c801a7ca4f22ca6
Sep 28 04:50:51 dist-test-slave-jccs krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:51.424501 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.416800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55811 (local address 127.0.98.62:40305)
0928 04:50:51.416876 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:51.416878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:51.417041 (+ 163us) server_negotiation.cc:407] Connection header received
0928 04:50:51.417517 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:51.417518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:51.417528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:51.417613 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:51.418682 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.419206 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.420109 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.420295 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.420345 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.422247 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:51.422249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:51.422249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:51.422255 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:51.423562 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:51.423851 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:51.423851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:51.423852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:51.423896 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:51.424072 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:51.424073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:51.424073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:51.424166 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:51.424210 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:51.424393 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 04:50:51.424444 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 04:50:51.424649 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.416769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:55811)
0928 04:50:51.416934 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:51.416965 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:51.417440 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:51.417660 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:51.417663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:51.418322 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:51.418609 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.418615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.419282 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.419283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.420027 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.420028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.420362 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.420363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.420450 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.420836 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:51.420843 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:51.422172 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:51.423624 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:51.423628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:51.423629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:51.423796 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:51.423947 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:51.423947 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:51.423947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:51.424026 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:51.424213 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:51.424216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:51.424303 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 04:50:51.424371 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:50:51.424462 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260928 04:50:51.425096 14594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40305
I20260928 04:50:51.425199 14594 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:51.425348 14594 heartbeater.cc:511] Master 127.0.98.62:40305 requested a full tablet report, sending...
I20260928 04:50:51.425800 14135 ts_manager.cc:194] Registered new tserver with Master: 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703)
I20260928 04:50:51.426106 14135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55811
I20260928 04:50:51.429560 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:50:51.436105 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.430863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.0.98.62:40305)
0928 04:50:51.430923 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:51.430925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:51.431084 (+ 159us) server_negotiation.cc:407] Connection header received
0928 04:50:51.431121 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:51.431121 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:51.431129 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:51.431173 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:51.431672 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.432111 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.432754 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.432937 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.432980 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.434097 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:51.434098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:51.434099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:51.434104 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:51.435146 (+ 1042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:51.435437 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:51.435438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:51.435440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:51.435505 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:51.435723 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:51.435723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:51.435724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:51.435832 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:51.435909 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:51.435974 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 04:50:51.436027 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:50:51.437453 14135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:50:51.438911 14135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:50:51.444677 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.440698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43703 (local address 127.0.0.1:60726)
0928 04:50:51.440805 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:51.440840 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:51.440898 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:51.441365 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:51.441367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:51.441370 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:51.441423 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.441424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.442447 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.442448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.443575 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.443577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.444380 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.444381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.444456 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.444468 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 04:50:51.444545 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 04:50:51.444598 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":66,"threads_started":1}
I20260928 04:50:51.444980 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.441013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.98.3:43703)
0928 04:50:51.441201 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:51.441206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:51.441216 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:50:51.441251 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:51.441253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:51.441267 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:51.441344 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:51.441551 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.442350 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.443669 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.444299 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.444381 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.444435 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:51.444872 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 04:50:51.444916 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260928 04:50:51.445248 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.441205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.0.98.2:44437)
0928 04:50:51.441358 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:51.441363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:51.441372 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:50:51.441411 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:51.441415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:51.441429 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:51.441641 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:51.441744 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.442798 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.444064 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.444736 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.444913 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.444967 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:51.445110 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 04:50:51.445150 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260928 04:50:51.445264 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.441017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.0.98.1:36883)
0928 04:50:51.441237 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:51.441241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:51.441454 (+ 213us) server_negotiation.cc:407] Connection header received
0928 04:50:51.441524 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:51.441526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:51.441541 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:51.441612 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:51.441838 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.442908 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.443931 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.444631 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.444700 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.444765 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:51.445114 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 04:50:51.445154 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
I20260928 04:50:51.444994 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.440878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36883 (local address 127.0.0.1:50202)
0928 04:50:51.441393 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:51.441402 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:51.441458 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:51.441726 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:51.441727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:51.441729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:51.441771 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.441772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.442987 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.442988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.443843 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.443845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.444701 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.444702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.444860 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.444865 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:50:51.444910 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:50:51.444940 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":66,"threads_started":1}
I20260928 04:50:51.445561 14528 tablet_service.cc:1467] Processing CreateTablet for tablet 8d626fee3e4b49afa40e58eea57745b6 (DEFAULT_TABLE table=test-table [id=6aa51a1aca1245fb931029f82ea59b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:50:51.445891 14528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d626fee3e4b49afa40e58eea57745b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:51.446012 14260 tablet_service.cc:1467] Processing CreateTablet for tablet 8d626fee3e4b49afa40e58eea57745b6 (DEFAULT_TABLE table=test-table [id=6aa51a1aca1245fb931029f82ea59b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:50:51.446362 14260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d626fee3e4b49afa40e58eea57745b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:51.448266 14608 tablet_bootstrap.cc:492] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Bootstrap starting.
I20260928 04:50:51.448403 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.440737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44437 (local address 127.0.0.1:49950)
0928 04:50:51.440995 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:51.441005 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:51.441053 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:51.441519 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:51.441520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:51.441522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:51.441570 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.441571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.442902 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.442904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.443989 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.443991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.444921 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.444922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.444979 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.444982 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:50:51.448283 (+ 3301us) client_negotiation.cc:240] Negotiation successful
0928 04:50:51.448330 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":58,"threads_started":1}
I20260928 04:50:51.448910 14609 tablet_bootstrap.cc:492] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Bootstrap starting.
I20260928 04:50:51.449010 14608 tablet_bootstrap.cc:654] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:51.449647 14609 tablet_bootstrap.cc:654] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:51.449630 14608 log.cc:824] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:51.449676 14394 tablet_service.cc:1467] Processing CreateTablet for tablet 8d626fee3e4b49afa40e58eea57745b6 (DEFAULT_TABLE table=test-table [id=6aa51a1aca1245fb931029f82ea59b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:50:51.449970 14394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d626fee3e4b49afa40e58eea57745b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:51.450140 14609 log.cc:824] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:51.450718 14609 tablet_bootstrap.cc:492] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: No bootstrap required, opened a new log
I20260928 04:50:51.450786 14609 ts_tablet_manager.cc:1397] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:50:51.451974 14612 tablet_bootstrap.cc:492] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: Bootstrap starting.
I20260928 04:50:51.452342 14608 tablet_bootstrap.cc:492] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: No bootstrap required, opened a new log
I20260928 04:50:51.452390 14608 ts_tablet_manager.cc:1397] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 04:50:51.452379 14609 raft_consensus.cc:348] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:51.452534 14609 raft_consensus.cc:374] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:51.452596 14609 raft_consensus.cc:729] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317807aa99a7476d954a352fcf1a075e, State: Initialized, Role: FOLLOWER
I20260928 04:50:51.452713 14609 consensus_queue.cc:260] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:51.452791 14612 tablet_bootstrap.cc:654] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:51.452988 14609 ts_tablet_manager.cc:1428] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:50:51.453138 14326 heartbeater.cc:503] Master 127.0.98.62:40305 was elected leader, sending a full tablet report...
I20260928 04:50:51.453327 14612 log.cc:824] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:51.453963 14612 tablet_bootstrap.cc:492] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: No bootstrap required, opened a new log
I20260928 04:50:51.454041 14612 ts_tablet_manager.cc:1397] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:50:51.454290 14608 raft_consensus.cc:348] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:51.454452 14608 raft_consensus.cc:374] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:51.454489 14608 raft_consensus.cc:729] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bcf7ba63ea24d0fa23d1a948b79f6dd, State: Initialized, Role: FOLLOWER
I20260928 04:50:51.454591 14608 consensus_queue.cc:260] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:51.454844 14608 ts_tablet_manager.cc:1428] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 04:50:51.454876 14594 heartbeater.cc:503] Master 127.0.98.62:40305 was elected leader, sending a full tablet report...
I20260928 04:50:51.455724 14612 raft_consensus.cc:348] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:51.455837 14612 raft_consensus.cc:374] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:51.455856 14612 raft_consensus.cc:729] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170cf5c796ff4af9a9e295aab2087cc3, State: Initialized, Role: FOLLOWER
I20260928 04:50:51.455951 14612 consensus_queue.cc:260] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:51.456183 14460 heartbeater.cc:503] Master 127.0.98.62:40305 was elected leader, sending a full tablet report...
I20260928 04:50:51.456183 14612 ts_tablet_manager.cc:1428] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 04:50:51.526758 14461 tablet.cc:2369] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:50:51.634236 14327 tablet.cc:2369] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:50:51.666721 14595 tablet.cc:2369] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:50:51.724773 14615 raft_consensus.cc:482] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:50:51.724895 14615 raft_consensus.cc:504] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:51.725260 14615 leader_election.cc:288] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 317807aa99a7476d954a352fcf1a075e (127.0.98.1:36883), 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437)
I20260928 04:50:51.728988 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.725513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40671 (local address 127.0.98.1:36883)
0928 04:50:51.725562 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:51.725565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:51.725677 (+ 112us) server_negotiation.cc:407] Connection header received
0928 04:50:51.725709 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:51.725710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:51.725721 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:51.725779 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:51.726004 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.726746 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.728266 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.728671 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.728737 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.728747 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:51.728908 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 04:50:51.728937 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 04:50:51.729033 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.725438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36883 (local address 127.0.98.3:40671)
0928 04:50:51.725606 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:51.725614 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:51.725662 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:51.725789 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:51.725790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:51.725791 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:51.725922 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.725924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.726828 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.726829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.728145 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.728148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.728742 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.728743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.728831 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.728836 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:50:51.728896 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 04:50:51.728935 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":105,"thread_start_us":88,"threads_started":1}
I20260928 04:50:51.729337 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.725427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44437 (local address 127.0.98.3:43769)
0928 04:50:51.725515 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:51.725526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:51.725573 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:51.725787 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:51.725788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:51.725789 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:51.725922 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.725924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.726832 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.726832 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.728145 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:51.728148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.729055 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.729056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:51.729142 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.729145 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:50:51.729233 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 04:50:51.729277 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":96}
I20260928 04:50:51.729398 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.725507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43769 (local address 127.0.98.2:44437)
0928 04:50:51.725575 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:51.725577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:51.725586 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:50:51.725664 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:51.725665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:51.725676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:51.725746 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:51.726004 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.726746 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.728256 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.728972 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.729036 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.729046 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:51.729283 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 04:50:51.729324 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 04:50:51.729535 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d626fee3e4b49afa40e58eea57745b6" candidate_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "317807aa99a7476d954a352fcf1a075e" is_pre_election: true
I20260928 04:50:51.729665 14414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d626fee3e4b49afa40e58eea57745b6" candidate_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170cf5c796ff4af9a9e295aab2087cc3" is_pre_election: true
I20260928 04:50:51.729717 14280 raft_consensus.cc:2454] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bcf7ba63ea24d0fa23d1a948b79f6dd in term 0.
I20260928 04:50:51.729835 14414 raft_consensus.cc:2454] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bcf7ba63ea24d0fa23d1a948b79f6dd in term 0.
I20260928 04:50:51.729987 14481 leader_election.cc:302] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 317807aa99a7476d954a352fcf1a075e, 8bcf7ba63ea24d0fa23d1a948b79f6dd; no voters:
I20260928 04:50:51.730093 14615 raft_consensus.cc:2781] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:50:51.730131 14615 raft_consensus.cc:482] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:50:51.730155 14615 raft_consensus.cc:3037] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:51.730885 14615 raft_consensus.cc:504] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:51.731032 14615 leader_election.cc:288] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [CANDIDATE]: Term 1 election: Requested vote from peers 317807aa99a7476d954a352fcf1a075e (127.0.98.1:36883), 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437)
I20260928 04:50:51.731190 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d626fee3e4b49afa40e58eea57745b6" candidate_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "317807aa99a7476d954a352fcf1a075e"
I20260928 04:50:51.731226 14414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d626fee3e4b49afa40e58eea57745b6" candidate_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170cf5c796ff4af9a9e295aab2087cc3"
I20260928 04:50:51.731283 14280 raft_consensus.cc:3037] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:51.731324 14414 raft_consensus.cc:3037] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:51.731945 14280 raft_consensus.cc:2454] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bcf7ba63ea24d0fa23d1a948b79f6dd in term 1.
I20260928 04:50:51.732050 14414 raft_consensus.cc:2454] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bcf7ba63ea24d0fa23d1a948b79f6dd in term 1.
I20260928 04:50:51.732158 14481 leader_election.cc:302] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 317807aa99a7476d954a352fcf1a075e, 8bcf7ba63ea24d0fa23d1a948b79f6dd; no voters:
I20260928 04:50:51.732229 14615 raft_consensus.cc:2781] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:50:51.732380 14615 raft_consensus.cc:686] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 1 LEADER]: Becoming Leader. State: Replica: 8bcf7ba63ea24d0fa23d1a948b79f6dd, State: Running, Role: LEADER
I20260928 04:50:51.732470 14615 consensus_queue.cc:237] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:51.733309 14135 catalog_manager.cc:5495] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } health_report { overall_health: UNKNOWN } } }
I20260928 04:50:51.796453 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:51.793975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.0.98.3:43703)
0928 04:50:51.794056 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:51.794059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:51.794067 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:50:51.794106 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:51.794107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:51.794136 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:51.794215 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:50:51.794372 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.794850 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.795621 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:51.795841 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:51.795915 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:51.796093 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:50:51.796213 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:50:51.796303 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:51.796343 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 04:50:51.796365 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 04:50:51.798411 14414 raft_consensus.cc:1261] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3 [term 1 FOLLOWER]: Refusing update from remote peer 8bcf7ba63ea24d0fa23d1a948b79f6dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:50:51.798471 14280 raft_consensus.cc:1261] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Refusing update from remote peer 8bcf7ba63ea24d0fa23d1a948b79f6dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:50:51.798761 14615 consensus_queue.cc:1035] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:50:51.798931 14618 consensus_queue.cc:1035] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:50:51.801106 14621 mvcc.cc:205] Tried to move back new op lower bound from 7334179028163518464 to 7334179027896700928. Current Snapshot: MvccSnapshot[applied={T|T < 7334179028163518464}]
I20260928 04:50:51.802076 14622 mvcc.cc:205] Tried to move back new op lower bound from 7334179028163518464 to 7334179027896700928. Current Snapshot: MvccSnapshot[applied={T|T < 7334179028163518464 or (T in {7334179028163518464})}]
I20260928 04:50:51.804098 14134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:50:51.804203 14134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:50:51.804945 14134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Sending DeleteTablet for 3 replicas of tablet 8d626fee3e4b49afa40e58eea57745b6
I20260928 04:50:51.805298 14394 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d626fee3e4b49afa40e58eea57745b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:50:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49950
I20260928 04:50:51.805313 14260 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d626fee3e4b49afa40e58eea57745b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:50:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50202
I20260928 04:50:51.805342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 14197
I20260928 04:50:51.806191 14528 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d626fee3e4b49afa40e58eea57745b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:50:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60726
I20260928 04:50:51.806247 14631 tablet_replica.cc:330] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: stopping tablet replica
I20260928 04:50:51.806349 14631 raft_consensus.cc:2229] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:50:51.806447 14631 raft_consensus.cc:2258] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:50:51.806598 14632 tablet_replica.cc:330] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: stopping tablet replica
I20260928 04:50:51.806690 14632 raft_consensus.cc:2229] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:50:51.806807 14632 raft_consensus.cc:2258] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:50:51.806918 14631 ts_tablet_manager.cc:1905] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:50:51.807188 14632 ts_tablet_manager.cc:1905] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:50:51.808885 14632 ts_tablet_manager.cc:1918] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:50:51.808887 14631 ts_tablet_manager.cc:1918] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:50:51.808979 14631 log.cc:1196] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/8d626fee3e4b49afa40e58eea57745b6
I20260928 04:50:51.808981 14632 log.cc:1196] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/8d626fee3e4b49afa40e58eea57745b6
I20260928 04:50:51.809291 14631 ts_tablet_manager.cc:1939] T 8d626fee3e4b49afa40e58eea57745b6 P 170cf5c796ff4af9a9e295aab2087cc3: Deleting consensus metadata
I20260928 04:50:51.809309 14632 ts_tablet_manager.cc:1939] T 8d626fee3e4b49afa40e58eea57745b6 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Deleting consensus metadata
I20260928 04:50:51.809661 14119 catalog_manager.cc:4841] TS 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703): tablet 8d626fee3e4b49afa40e58eea57745b6 (table test-table [id=6aa51a1aca1245fb931029f82ea59b53]) successfully deleted
I20260928 04:50:51.809800 14120 catalog_manager.cc:4841] TS 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437): tablet 8d626fee3e4b49afa40e58eea57745b6 (table test-table [id=6aa51a1aca1245fb931029f82ea59b53]) successfully deleted
W20260928 04:50:51.810531 14119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36883 (error 108)
W20260928 04:50:51.811501 14604 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:50:51.810886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36883 (local address 127.0.0.1:50218)
0928 04:50:51.810945 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:51.810956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:51.810993 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:51.811449 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36883: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 04:50:51.811622 14119 catalog_manager.cc:4584] TS 317807aa99a7476d954a352fcf1a075e (127.0.98.1:36883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d626fee3e4b49afa40e58eea57745b6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36883: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:50:51.811677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 14331
I20260928 04:50:51.816437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 14465
I20260928 04:50:51.821322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 14103
I20260928 04:50:51.826514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40305
--webserver_interface=127.0.98.62
--webserver_port=41323
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:51.904632 14634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:51.904821 14634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:51.904860 14634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:51.906033 14634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:50:51.906073 14634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:51.906087 14634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:51.906100 14634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:50:51.906159 14634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:50:51.907513 14634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:51.907732 14634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:51.907940 14634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:51.916026 14643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:51.916181 14634 server_base.cc:1034] running on GCE node
W20260928 04:50:51.916028 14641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:51.916023 14640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:51.916543 14634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:51.916829 14634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:51.917970 14634 hybrid_clock.cc:648] HybridClock initialized: now 1790571051917957 us; error 37 us; skew 500 ppm
Sep 28 04:50:51 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:51.926316 14634 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:50:51.926812 14634 webserver.cc:460] Webserver started at http://127.0.98.62:41323/ using document root <none> and password file <none>
I20260928 04:50:51.927075 14634 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:51.927153 14634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:51.928416 14634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:51.929040 14650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:51.929230 14634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:50:51.929309 14634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "6dcb8e1046d94b768c5713bf11e18f27"
format_stamp: "Formatted at 2026-09-28 04:50:50 on dist-test-slave-jccs"
server_key: "f58d2f938d63880f7add844bf355e95a"
server_key_iv: "a12c9f7dcfb3993bf19d3f6e9a20358b"
server_key_version: "encryptionkey@0"
I20260928 04:50:51.929716 14634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:51.947525 14634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:51.949733 14634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:51.949849 14634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:51.953604 14634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40305
I20260928 04:50:51.953663 14702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40305 every 8 connection(s)
I20260928 04:50:51.953979 14634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:50:51.955348 14703 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:50:51.955863 14703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Bootstrap starting.
I20260928 04:50:51.961151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 14634
I20260928 04:50:51.961503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36883
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40305
--webserver_port=38331
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:50:51.964262 14703 log.cc:824] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:51.966125 14703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:50:51.966396 14703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Bootstrap complete.
I20260928 04:50:51.968708 14703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:51.968930 14703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcb8e1046d94b768c5713bf11e18f27, State: Initialized, Role: FOLLOWER
I20260928 04:50:51.969094 14703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:51.969152 14703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:50:51.969235 14703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:50:51.969292 14703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:50:51.970281 14703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:51.970366 14703 leader_election.cc:302] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dcb8e1046d94b768c5713bf11e18f27; no voters:
I20260928 04:50:51.970542 14703 leader_election.cc:288] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:50:51.970573 14707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:50:51.970685 14707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 2 LEADER]: Becoming Leader. State: Replica: 6dcb8e1046d94b768c5713bf11e18f27, State: Running, Role: LEADER
I20260928 04:50:51.970764 14707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:51.970795 14703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:50:51.971064 14708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6dcb8e1046d94b768c5713bf11e18f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } } }
I20260928 04:50:51.971061 14707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dcb8e1046d94b768c5713bf11e18f27. Latest consensus state: current_term: 2 leader_uuid: "6dcb8e1046d94b768c5713bf11e18f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } } }
I20260928 04:50:51.971267 14707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:51.971237 14708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:51.971796 14714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:50:51.972599 14714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:50:51.972697 14714 catalog_manager.cc:1259] Loaded cluster ID: 83a7b8c4122f4e0f9d1b1a07f4620c70
I20260928 04:50:51.972728 14714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:50:51.972949 14714 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 04:50:51.977478 14714 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 04:50:51.980218 14714 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:50:51.980405 14714 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Loaded TSK: 0
I20260928 04:50:51.980611 14714 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:50:52.040771 14705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:52.040982 14705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:52.041034 14705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:52.042266 14705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:52.042328 14705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:52.042373 14705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:50:52.043780 14705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38331
--tserver_master_addrs=127.0.98.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:52.044013 14705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:52.044210 14705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:52.049578 14730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:52.049628 14731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:52.049644 14733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:52.049921 14705 server_base.cc:1034] running on GCE node
I20260928 04:50:52.050104 14705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:52.050331 14705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:52.051479 14705 hybrid_clock.cc:648] HybridClock initialized: now 1790571052051463 us; error 32 us; skew 500 ppm
Sep 28 04:50:52 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:52.053848 14705 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:50:52.054304 14705 webserver.cc:460] Webserver started at http://127.0.98.1:38331/ using document root <none> and password file <none>
I20260928 04:50:52.054570 14705 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:52.054622 14705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:52.055917 14705 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:52.056567 14740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.056710 14705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:50:52.056789 14705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "317807aa99a7476d954a352fcf1a075e"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "c00963a0ae8d63d570e0224beaa54359"
server_key_iv: "945e1ce0a2f5fdbbeea7dd27e47d3c30"
server_key_version: "encryptionkey@0"
I20260928 04:50:52.057089 14705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:52.077504 14705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:52.079553 14705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:52.079701 14705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:52.079932 14705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:52.080365 14747 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:50:52.082050 14705 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:50:52.082108 14705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:50:52.082160 14705 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:50:52.082760 14705 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:50:52.082830 14705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:50:52.082891 14747 tablet_bootstrap.cc:492] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Bootstrap starting.
I20260928 04:50:52.089008 14705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36883
I20260928 04:50:52.089135 14854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36883 every 8 connection(s)
I20260928 04:50:52.089427 14705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:50:52.093474 14747 log.cc:824] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:52.095765 14747 tablet_bootstrap.cc:492] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:50:52.096037 14747 tablet_bootstrap.cc:492] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Bootstrap complete.
I20260928 04:50:52.096791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 14705
I20260928 04:50:52.097090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44437
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40305
--webserver_port=34837
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:50:52.097213 14747 ts_tablet_manager.cc:1397] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.005s
I20260928 04:50:52.098803 14747 raft_consensus.cc:348] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:52.099118 14747 raft_consensus.cc:729] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317807aa99a7476d954a352fcf1a075e, State: Initialized, Role: FOLLOWER
I20260928 04:50:52.099263 14747 consensus_queue.cc:260] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } } peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } }
I20260928 04:50:52.099537 14747 ts_tablet_manager.cc:1428] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
Sep 28 04:50:52 dist-test-slave-jccs krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:52.103484 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.090116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58089 (local address 127.0.98.62:40305)
0928 04:50:52.093416 (+ 3300us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.093424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.093465 (+ 41us) server_negotiation.cc:407] Connection header received
0928 04:50:52.094198 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.094203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.094244 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.094362 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:52.094956 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.095784 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.097031 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.097667 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.097751 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.100342 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:52.100346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:52.100347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:52.100358 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:52.101864 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:52.102291 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:52.102295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:52.102295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:52.102382 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:52.102634 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:52.102635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:52.102636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:52.102832 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:52.102913 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.103197 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.103322 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3256,"thread_start_us":88,"threads_started":1}
I20260928 04:50:52.103484 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.090060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:58089)
0928 04:50:52.093293 (+ 3233us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.093337 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:52.093988 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:52.094387 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:52.094391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:52.094649 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:52.094812 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.094816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.095928 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.095929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.096914 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.096917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.097880 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.097881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.098033 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.098623 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:52.098634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:52.100259 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:52.101968 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:52.101974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:52.101976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:52.102225 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:52.102458 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:52.102459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:52.102459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:52.102564 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:52.102907 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:52.102913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:52.103044 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 04:50:52.103147 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 04:50:52.103312 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3194,"thread_start_us":111,"threads_started":1}
I20260928 04:50:52.104110 14855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40305
I20260928 04:50:52.104192 14855 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:52.104348 14855 heartbeater.cc:511] Master 127.0.98.62:40305 requested a full tablet report, sending...
I20260928 04:50:52.105036 14667 ts_manager.cc:194] Registered new tserver with Master: 317807aa99a7476d954a352fcf1a075e (127.0.98.1:36883)
I20260928 04:50:52.106076 14667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58089
I20260928 04:50:52.107055 14855 heartbeater.cc:503] Master 127.0.98.62:40305 was elected leader, sending a full tablet report...
I20260928 04:50:52.110225 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.105612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36883 (local address 127.0.0.1:50226)
0928 04:50:52.106651 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.106686 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:52.106771 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:52.107029 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:52.107031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:52.107034 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:52.107100 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.107102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.108346 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.108347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.109118 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.109120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.110022 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.110023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.110079 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.110088 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:50:52.110136 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 04:50:52.110171 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":74,"threads_started":1}
I20260928 04:50:52.110257 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.106367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.0.98.1:36883)
0928 04:50:52.106518 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.106522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.106772 (+ 250us) server_negotiation.cc:407] Connection header received
0928 04:50:52.106867 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.106869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.106919 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.107004 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:52.107194 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.108266 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.109232 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.109949 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.110023 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.110071 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.110159 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.110191 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260928 04:50:52.110736 14789 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d626fee3e4b49afa40e58eea57745b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:50:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50226
I20260928 04:50:52.111187 14864 tablet_replica.cc:330] stopping tablet replica
I20260928 04:50:52.111282 14864 raft_consensus.cc:2229] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:50:52.111342 14864 raft_consensus.cc:2258] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:50:52.111613 14864 ts_tablet_manager.cc:1905] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:50:52.113734 14864 ts_tablet_manager.cc:1918] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:50:52.113790 14864 log.cc:1196] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/8d626fee3e4b49afa40e58eea57745b6
I20260928 04:50:52.113996 14864 ts_tablet_manager.cc:1939] T 8d626fee3e4b49afa40e58eea57745b6 P 317807aa99a7476d954a352fcf1a075e: Deleting consensus metadata
I20260928 04:50:52.114431 14651 catalog_manager.cc:4841] TS 317807aa99a7476d954a352fcf1a075e (127.0.98.1:36883): tablet 8d626fee3e4b49afa40e58eea57745b6 (table test-table [id=6aa51a1aca1245fb931029f82ea59b53]) successfully deleted
W20260928 04:50:52.179312 14860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:52.179538 14860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:52.179575 14860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:52.180688 14860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:52.180749 14860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:52.180792 14860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:50:52.182221 14860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34837
--tserver_master_addrs=127.0.98.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:52.182468 14860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:52.182693 14860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:52.187381 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:52.187359 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:52.187348 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:52.187389 14860 server_base.cc:1034] running on GCE node
I20260928 04:50:52.187691 14860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:52.187947 14860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:52.189108 14860 hybrid_clock.cc:648] HybridClock initialized: now 1790571052189080 us; error 42 us; skew 500 ppm
Sep 28 04:50:52 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:52.191377 14860 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:50:52.191756 14860 webserver.cc:460] Webserver started at http://127.0.98.2:34837/ using document root <none> and password file <none>
I20260928 04:50:52.191962 14860 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:52.192039 14860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:52.193295 14860 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:52.193856 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.194010 14860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:50:52.194098 14860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "170cf5c796ff4af9a9e295aab2087cc3"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "ba4058e75d09567d3af0015086aae8ca"
server_key_iv: "32ac7d0ec544c600d19c4d971510ae0a"
server_key_version: "encryptionkey@0"
I20260928 04:50:52.194409 14860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:52.203296 14860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:52.204999 14860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:52.205152 14860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:52.205415 14860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:52.205695 14860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:52.205747 14860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.205793 14860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:52.205822 14860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.211350 14860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44437
I20260928 04:50:52.211410 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44437 every 8 connection(s)
I20260928 04:50:52.211720 14860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:50:52.213788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 14860
I20260928 04:50:52.214043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43703
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40305
--webserver_port=41463
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:50:52 dist-test-slave-jccs krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:52.222026 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.212337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:37539)
0928 04:50:52.212462 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.212502 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:52.212984 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:52.213251 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:52.213255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:52.213805 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:52.214039 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.214043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.215021 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.215022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.216243 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.216246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.216597 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.216598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.216749 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.217253 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:52.217262 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:52.218783 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:52.220681 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:52.220686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:52.220688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:52.220901 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:52.221108 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:52.221109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:52.221109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:52.221257 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:52.221558 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:52.221563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:52.221667 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 04:50:52.221737 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:50:52.221878 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":64,"threads_started":1}
I20260928 04:50:52.222052 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.212359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37539 (local address 127.0.98.62:40305)
0928 04:50:52.212397 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.212399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.212580 (+ 181us) server_negotiation.cc:407] Connection header received
0928 04:50:52.213101 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.213102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.213117 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.213195 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:52.214156 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.214919 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.216332 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.216521 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.216579 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.218881 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:52.218883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:52.218884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:52.218893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:52.220594 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:52.220974 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:52.220976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:52.220976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:52.221039 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:52.221329 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:52.221330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:52.221330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:52.221484 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:52.221558 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.221828 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.221924 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:50:52.222714 14993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40305
I20260928 04:50:52.222829 14993 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:52.223022 14993 heartbeater.cc:511] Master 127.0.98.62:40305 requested a full tablet report, sending...
I20260928 04:50:52.223417 14667 ts_manager.cc:194] Registered new tserver with Master: 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437)
I20260928 04:50:52.223953 14667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37539
W20260928 04:50:52.295305 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:52.295492 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:52.295516 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:52.296640 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:52.296677 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:52.296700 14996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:50:52.298123 14996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41463
--tserver_master_addrs=127.0.98.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:52.298352 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:52.298561 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:50:52.303463 14996 server_base.cc:1034] running on GCE node
W20260928 04:50:52.303426 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:52.303440 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:52.303424 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:52.303814 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:52.304066 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:52.305234 14996 hybrid_clock.cc:648] HybridClock initialized: now 1790571052305223 us; error 24 us; skew 500 ppm
Sep 28 04:50:52 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:52.307523 14996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:50:52.307948 14996 webserver.cc:460] Webserver started at http://127.0.98.3:41463/ using document root <none> and password file <none>
I20260928 04:50:52.308157 14996 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:52.308254 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:52.309573 14996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:52.310119 15011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.310294 14996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:50:52.310389 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "f405870892a3503356aa30568ed8068c"
server_key_iv: "46dc594d2eb06ca92f66063b59328112"
server_key_version: "encryptionkey@0"
I20260928 04:50:52.310736 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:52.324438 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:52.326274 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:52.326427 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:52.326675 14996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:52.326962 14996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:52.327015 14996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.327057 14996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:52.327085 14996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.333065 14996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43703
I20260928 04:50:52.333173 15124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43703 every 8 connection(s)
I20260928 04:50:52.333514 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:50:52 dist-test-slave-jccs krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:50:52.339496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 14996
I20260928 04:50:52.342473 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.334176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40899 (local address 127.0.98.62:40305)
0928 04:50:52.334218 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.334220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.334538 (+ 318us) server_negotiation.cc:407] Connection header received
0928 04:50:52.335007 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.335008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.335018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.335064 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:52.335922 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.336385 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.337373 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.337546 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.337606 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.339648 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:52.339649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:52.339649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:52.339655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:52.341157 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:52.341598 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:52.341599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:52.341600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:52.341664 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:52.341893 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:52.341894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:52.341894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:52.342021 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:52.342090 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.342344 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.342412 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:50:52.342851 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.334181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:40899)
0928 04:50:52.334364 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.334421 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:52.334920 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:52.335122 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:52.335125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:52.335669 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:52.335849 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.335853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.336459 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.336460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.337304 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.337306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.337641 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.337641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.337733 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.338141 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:52.338148 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:52.339530 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:52.341290 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:52.341297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:52.341299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:52.341534 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:52.341732 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:52.341733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:52.341733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:52.341832 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:52.342084 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:52.342088 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:52.342206 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 04:50:52.342314 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 04:50:52.342455 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260928 04:50:52.343299 15125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40305
I20260928 04:50:52.343393 15125 heartbeater.cc:464] Registering TS with master...
I20260928 04:50:52.343585 15125 heartbeater.cc:511] Master 127.0.98.62:40305 requested a full tablet report, sending...
I20260928 04:50:52.343978 14667 ts_manager.cc:194] Registered new tserver with Master: 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703)
I20260928 04:50:52.344451 14667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40899
I20260928 04:50:52.346395 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.339871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.0.98.62:40305)
0928 04:50:52.340020 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.340022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.340031 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:50:52.340111 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.340111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.340122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.340200 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:52.340745 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.341278 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.342230 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.342644 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.342710 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.344103 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:52.344104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:52.344105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:52.344111 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:52.345567 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:52.345856 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:52.345856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:52.345856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:52.345901 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:52.346051 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:52.346052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:52.346052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:52.346139 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:52.346238 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.346284 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.346338 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260928 04:50:52.347098 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:50:52.349844 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.347415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.0.98.62:40305)
0928 04:50:52.347477 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.347478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.347548 (+ 70us) server_negotiation.cc:407] Connection header received
0928 04:50:52.347621 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.347622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.347653 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.347738 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:50:52.347830 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.348271 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.349024 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.349308 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.349371 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.349542 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:50:52.349642 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:50:52.349735 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.349755 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.349774 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:50:52.350136 14667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:50:52.351774 14667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:50:52.353914 14789 tablet_service.cc:1467] Processing CreateTablet for tablet 8db864a6f96d45028e98c26e6ef14717 (DEFAULT_TABLE table=test-table [id=9c18f74c667f41428623f29d9e92a4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:50:52.354110 14789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db864a6f96d45028e98c26e6ef14717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:52.355449 14747 tablet_bootstrap.cc:492] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: Bootstrap starting.
I20260928 04:50:52.355772 14747 tablet_bootstrap.cc:654] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:52.356328 14747 tablet_bootstrap.cc:492] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: No bootstrap required, opened a new log
I20260928 04:50:52.356389 14747 ts_tablet_manager.cc:1397] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 04:50:52.356496 14747 raft_consensus.cc:348] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.356572 14747 raft_consensus.cc:374] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:52.356621 14747 raft_consensus.cc:729] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317807aa99a7476d954a352fcf1a075e, State: Initialized, Role: FOLLOWER
I20260928 04:50:52.356663 14747 consensus_queue.cc:260] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.356745 14747 ts_tablet_manager.cc:1428] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.357616 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.353538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44437 (local address 127.0.0.1:49960)
0928 04:50:52.353849 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.353858 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:52.353914 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:52.354124 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:52.354125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:52.354128 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:52.354164 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.354165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.355188 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.355189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.356176 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.356179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.357400 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.357401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.357463 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.357465 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:50:52.357510 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:50:52.357546 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":120,"thread_start_us":77,"threads_started":1}
I20260928 04:50:52.357832 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.353644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.0.98.2:44437)
0928 04:50:52.353800 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.353805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.353927 (+ 122us) server_negotiation.cc:407] Connection header received
0928 04:50:52.353967 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.353969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.353981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.354076 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:52.354250 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.355094 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.356282 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.357241 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.357286 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.357339 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.357715 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.357757 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260928 04:50:52.358687 14927 tablet_service.cc:1467] Processing CreateTablet for tablet 8db864a6f96d45028e98c26e6ef14717 (DEFAULT_TABLE table=test-table [id=9c18f74c667f41428623f29d9e92a4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:50:52.359046 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db864a6f96d45028e98c26e6ef14717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:52.359163 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.353538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43703 (local address 127.0.0.1:60740)
0928 04:50:52.353573 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.353582 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:52.353614 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:52.355680 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:52.355681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:52.355682 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:52.355718 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.355719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.357032 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.357033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.358048 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.358051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.358905 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.358906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.358973 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.358976 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:50:52.359051 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 04:50:52.359095 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 04:50:52.359243 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.353611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.0.98.3:43703)
0928 04:50:52.355513 (+ 1902us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.355517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.355528 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:50:52.355561 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.355563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.355576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.355677 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:52.355781 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.356738 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.358121 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.358771 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.358846 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.358898 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.359066 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.359156 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1869,"thread_start_us":78,"threads_started":1}
I20260928 04:50:52.359791 15059 tablet_service.cc:1467] Processing CreateTablet for tablet 8db864a6f96d45028e98c26e6ef14717 (DEFAULT_TABLE table=test-table [id=9c18f74c667f41428623f29d9e92a4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:50:52.360077 15059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db864a6f96d45028e98c26e6ef14717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:50:52.361902 15135 tablet_bootstrap.cc:492] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: Bootstrap starting.
I20260928 04:50:52.362617 15135 tablet_bootstrap.cc:654] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:52.362709 15136 tablet_bootstrap.cc:492] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Bootstrap starting.
I20260928 04:50:52.363231 15135 log.cc:824] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:52.363355 15136 tablet_bootstrap.cc:654] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Neither blocks nor log segments found. Creating new log.
I20260928 04:50:52.363911 15136 log.cc:824] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:52.364029 15135 tablet_bootstrap.cc:492] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: No bootstrap required, opened a new log
I20260928 04:50:52.364104 15135 ts_tablet_manager.cc:1397] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:50:52.364708 15136 tablet_bootstrap.cc:492] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: No bootstrap required, opened a new log
I20260928 04:50:52.364776 15136 ts_tablet_manager.cc:1397] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:50:52.366180 15135 raft_consensus.cc:348] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.366355 15135 raft_consensus.cc:374] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:52.366391 15135 raft_consensus.cc:729] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170cf5c796ff4af9a9e295aab2087cc3, State: Initialized, Role: FOLLOWER
I20260928 04:50:52.366513 15135 consensus_queue.cc:260] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.366773 15135 ts_tablet_manager.cc:1428] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:50:52.366780 14993 heartbeater.cc:503] Master 127.0.98.62:40305 was elected leader, sending a full tablet report...
I20260928 04:50:52.366776 15136 raft_consensus.cc:348] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.366942 15136 raft_consensus.cc:374] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:50:52.366993 15136 raft_consensus.cc:729] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bcf7ba63ea24d0fa23d1a948b79f6dd, State: Initialized, Role: FOLLOWER
I20260928 04:50:52.367129 15136 consensus_queue.cc:260] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.367391 15136 ts_tablet_manager.cc:1428] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:50:52.367413 15125 heartbeater.cc:503] Master 127.0.98.62:40305 was elected leader, sending a full tablet report...
W20260928 04:50:52.462250 14994 tablet.cc:2369] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:50:52.530381 15139 raft_consensus.cc:482] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:50:52.530499 15139 raft_consensus.cc:504] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.530814 15139 leader_election.cc:288] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703), 317807aa99a7476d954a352fcf1a075e (127.0.98.1:36883)
I20260928 04:50:52.533968 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.531297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53369 (local address 127.0.98.1:36883)
0928 04:50:52.531345 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.531347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.531456 (+ 109us) server_negotiation.cc:407] Connection header received
0928 04:50:52.531504 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.531506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.531517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.531578 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:52.531720 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.532237 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.533209 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.533582 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.533677 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.533686 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.533874 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.533913 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:50:52.533954 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.531088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43703 (local address 127.0.98.2:52953)
0928 04:50:52.531133 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.531145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:52.531192 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:52.531369 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:52.531370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:52.531372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:52.531468 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.531470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.532202 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.532203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.533048 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.533050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.533671 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.533672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.533748 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.533751 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:50:52.533810 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 04:50:52.533875 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 04:50:52.533963 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.531209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36883 (local address 127.0.98.2:53369)
0928 04:50:52.531369 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.531380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:52.531452 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:52.531587 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:52.531588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:52.531590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:50:52.531655 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.531656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.532311 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.532311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.533118 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:52.533121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.533701 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.533702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:52.533796 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.533800 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:50:52.533852 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 04:50:52.533891 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260928 04:50:52.534134 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.531158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52953 (local address 127.0.98.3:43703)
0928 04:50:52.531209 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.531211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.531223 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:50:52.531255 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.531256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.531268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.531366 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:50:52.531540 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.532134 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.533142 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.533558 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.533646 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.533657 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.534012 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.534056 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:50:52.534284 14801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db864a6f96d45028e98c26e6ef14717" candidate_uuid: "170cf5c796ff4af9a9e295aab2087cc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "317807aa99a7476d954a352fcf1a075e" is_pre_election: true
I20260928 04:50:52.534379 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db864a6f96d45028e98c26e6ef14717" candidate_uuid: "170cf5c796ff4af9a9e295aab2087cc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" is_pre_election: true
I20260928 04:50:52.534446 14801 raft_consensus.cc:2454] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170cf5c796ff4af9a9e295aab2087cc3 in term 0.
I20260928 04:50:52.534495 15079 raft_consensus.cc:2454] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170cf5c796ff4af9a9e295aab2087cc3 in term 0.
I20260928 04:50:52.534679 14880 leader_election.cc:302] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170cf5c796ff4af9a9e295aab2087cc3, 8bcf7ba63ea24d0fa23d1a948b79f6dd; no voters:
I20260928 04:50:52.534835 15139 raft_consensus.cc:2781] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:50:52.534914 15139 raft_consensus.cc:482] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:50:52.534930 15139 raft_consensus.cc:3037] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:52.535580 15139 raft_consensus.cc:504] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.535703 15139 leader_election.cc:288] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [CANDIDATE]: Term 1 election: Requested vote from peers 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703), 317807aa99a7476d954a352fcf1a075e (127.0.98.1:36883)
I20260928 04:50:52.535873 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db864a6f96d45028e98c26e6ef14717" candidate_uuid: "170cf5c796ff4af9a9e295aab2087cc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd"
I20260928 04:50:52.535907 14801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db864a6f96d45028e98c26e6ef14717" candidate_uuid: "170cf5c796ff4af9a9e295aab2087cc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "317807aa99a7476d954a352fcf1a075e"
I20260928 04:50:52.535938 15079 raft_consensus.cc:3037] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:52.535997 14801 raft_consensus.cc:3037] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:50:52.536638 15079 raft_consensus.cc:2454] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170cf5c796ff4af9a9e295aab2087cc3 in term 1.
I20260928 04:50:52.536806 14880 leader_election.cc:302] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170cf5c796ff4af9a9e295aab2087cc3, 8bcf7ba63ea24d0fa23d1a948b79f6dd; no voters:
I20260928 04:50:52.536902 15139 raft_consensus.cc:2781] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:50:52.537042 15139 raft_consensus.cc:686] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 1 LEADER]: Becoming Leader. State: Replica: 170cf5c796ff4af9a9e295aab2087cc3, State: Running, Role: LEADER
I20260928 04:50:52.537137 15139 consensus_queue.cc:237] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.537228 14801 raft_consensus.cc:2454] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170cf5c796ff4af9a9e295aab2087cc3 in term 1.
I20260928 04:50:52.538009 14667 catalog_manager.cc:5495] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "170cf5c796ff4af9a9e295aab2087cc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } health_report { overall_health: UNKNOWN } } }
I20260928 04:50:52.583132 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.581039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.0.98.2:44437)
0928 04:50:52.581085 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:52.581087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:52.581095 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:50:52.581199 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:52.581200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:52.581222 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:52.581285 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:50:52.581390 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.581791 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.582494 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:52.582668 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:52.582716 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:52.582841 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:50:52.582939 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:50:52.583009 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:52.583043 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 04:50:52.583060 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 04:50:52.584097 15126 tablet.cc:2369] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:50:52.584915 15079 raft_consensus.cc:1261] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 1 FOLLOWER]: Refusing update from remote peer 170cf5c796ff4af9a9e295aab2087cc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:50:52.584920 14801 raft_consensus.cc:1261] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Refusing update from remote peer 170cf5c796ff4af9a9e295aab2087cc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:50:52.585283 15139 consensus_queue.cc:1035] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:50:52.585412 15142 consensus_queue.cc:1035] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:50:52.587582 15147 mvcc.cc:205] Tried to move back new op lower bound from 7334179031384854528 to 7334179031192563712. Current Snapshot: MvccSnapshot[applied={T|T < 7334179031384854528}]
I20260928 04:50:52.588238 15146 mvcc.cc:205] Tried to move back new op lower bound from 7334179031384854528 to 7334179031192563712. Current Snapshot: MvccSnapshot[applied={T|T < 7334179031384854528}]
I20260928 04:50:52.589291 14666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:50:52.589390 14666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:50:52.590126 14666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Sending DeleteTablet for 3 replicas of tablet 8db864a6f96d45028e98c26e6ef14717
I20260928 04:50:52.590416 15059 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db864a6f96d45028e98c26e6ef14717 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:50:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60740
I20260928 04:50:52.590441 14927 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db864a6f96d45028e98c26e6ef14717 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:50:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49960
I20260928 04:50:52.590571 14789 tablet_service.cc:1514] Processing DeleteTablet for tablet 8db864a6f96d45028e98c26e6ef14717 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:50:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50226
I20260928 04:50:52.590656 14864 tablet_replica.cc:330] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: stopping tablet replica
I20260928 04:50:52.590711 14864 raft_consensus.cc:2229] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:50:52.590755 14864 raft_consensus.cc:2258] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:50:52.590754 15154 tablet_replica.cc:330] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: stopping tablet replica
I20260928 04:50:52.590778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 14705
I20260928 04:50:52.590830 15154 raft_consensus.cc:2229] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:50:52.590916 15154 raft_consensus.cc:2258] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:50:52.590633 15155 tablet_replica.cc:330] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: stopping tablet replica
I20260928 04:50:52.591398 15155 raft_consensus.cc:2229] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:50:52.591452 15155 raft_consensus.cc:2258] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:50:52.591789 15155 ts_tablet_manager.cc:1905] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:50:52.593441 15155 ts_tablet_manager.cc:1918] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:50:52.593508 15155 log.cc:1196] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/8db864a6f96d45028e98c26e6ef14717
I20260928 04:50:52.593782 15155 ts_tablet_manager.cc:1939] T 8db864a6f96d45028e98c26e6ef14717 P 170cf5c796ff4af9a9e295aab2087cc3: Deleting consensus metadata
I20260928 04:50:52.594465 14652 catalog_manager.cc:4841] TS 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437): tablet 8db864a6f96d45028e98c26e6ef14717 (table test-table [id=9c18f74c667f41428623f29d9e92a4a7]) successfully deleted
W20260928 04:50:52.596760 14651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36883 (error 108)
I20260928 04:50:52.597093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 14860
I20260928 04:50:52.598191 15154 ts_tablet_manager.cc:1905] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:50:52.598598 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.597144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36883 (local address 127.0.0.1:50234)
0928 04:50:52.598519 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.598562 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1353}
W20260928 04:50:52.598798 14651 catalog_manager.cc:4584] TS 317807aa99a7476d954a352fcf1a075e (127.0.98.1:36883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8db864a6f96d45028e98c26e6ef14717: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36883: connect: Connection refused (error 111)
I20260928 04:50:52.599985 15154 ts_tablet_manager.cc:1918] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:50:52.600047 15154 log.cc:1196] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/8db864a6f96d45028e98c26e6ef14717
I20260928 04:50:52.600310 15154 ts_tablet_manager.cc:1939] T 8db864a6f96d45028e98c26e6ef14717 P 8bcf7ba63ea24d0fa23d1a948b79f6dd: Deleting consensus metadata
I20260928 04:50:52.600708 14651 catalog_manager.cc:4841] TS 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703): tablet 8db864a6f96d45028e98c26e6ef14717 (table test-table [id=9c18f74c667f41428623f29d9e92a4a7]) successfully deleted
I20260928 04:50:52.602303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 14996
I20260928 04:50:52.606544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 14634
I20260928 04:50:52.611544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40305
--webserver_interface=127.0.98.62
--webserver_port=41323
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:52.689474 15157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:52.689707 15157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:52.689744 15157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:52.690901 15157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:50:52.690970 15157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:52.690994 15157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:52.691013 15157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:50:52.691054 15157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:50:52.692384 15157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:52.692636 15157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:52.692860 15157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:52.697479 15163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:52.697531 15165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:52.697669 15157 server_base.cc:1034] running on GCE node
W20260928 04:50:52.697476 15162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:52.697957 15157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:52.698264 15157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:52.699424 15157 hybrid_clock.cc:648] HybridClock initialized: now 1790571052699414 us; error 26 us; skew 500 ppm
Sep 28 04:50:52 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:52.701768 15157 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:50:52.702236 15157 webserver.cc:460] Webserver started at http://127.0.98.62:41323/ using document root <none> and password file <none>
I20260928 04:50:52.702464 15157 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:52.702533 15157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:52.703780 15157 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:52.704393 15172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.704558 15157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:50:52.704668 15157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "6dcb8e1046d94b768c5713bf11e18f27"
format_stamp: "Formatted at 2026-09-28 04:50:50 on dist-test-slave-jccs"
server_key: "f58d2f938d63880f7add844bf355e95a"
server_key_iv: "a12c9f7dcfb3993bf19d3f6e9a20358b"
server_key_version: "encryptionkey@0"
I20260928 04:50:52.705019 15157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:52.722432 15157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:52.724200 15157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:52.724371 15157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:52.728024 15157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40305
I20260928 04:50:52.728058 15224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40305 every 8 connection(s)
I20260928 04:50:52.728447 15157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:50:52.730050 15225 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:50:52.730674 15225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Bootstrap starting.
I20260928 04:50:52.736012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 15157
I20260928 04:50:52.736303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36883
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40305
--webserver_port=38331
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:50:52.738487 15225 log.cc:824] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:52.740561 15225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:50:52.740842 15225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27: Bootstrap complete.
I20260928 04:50:52.743018 15225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:52.743224 15225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcb8e1046d94b768c5713bf11e18f27, State: Initialized, Role: FOLLOWER
I20260928 04:50:52.743373 15225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:52.743433 15225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:50:52.743495 15225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:50:52.743536 15225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 2 FOLLOWER]: Advancing to term 3
I20260928 04:50:52.744405 15225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:52.744488 15225 leader_election.cc:302] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dcb8e1046d94b768c5713bf11e18f27; no voters:
I20260928 04:50:52.744644 15225 leader_election.cc:288] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 04:50:52.744686 15229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 04:50:52.744799 15229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [term 3 LEADER]: Becoming Leader. State: Replica: 6dcb8e1046d94b768c5713bf11e18f27, State: Running, Role: LEADER
I20260928 04:50:52.744880 15225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:50:52.744870 15229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } }
I20260928 04:50:52.745219 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dcb8e1046d94b768c5713bf11e18f27. Latest consensus state: current_term: 3 leader_uuid: "6dcb8e1046d94b768c5713bf11e18f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } } }
I20260928 04:50:52.745297 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:52.745422 15231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6dcb8e1046d94b768c5713bf11e18f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dcb8e1046d94b768c5713bf11e18f27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40305 } } }
I20260928 04:50:52.745540 15231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dcb8e1046d94b768c5713bf11e18f27 [sys.catalog]: This master's current role is: LEADER
I20260928 04:50:52.745842 15235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:50:52.746574 15235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:50:52.746829 15235 catalog_manager.cc:1259] Loaded cluster ID: 83a7b8c4122f4e0f9d1b1a07f4620c70
I20260928 04:50:52.746908 15235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 04:50:52.747098 15235 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4a06b099ad google::LogMessage::Fail() at ??:0
@ 0x7f4a06b0c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4a06b094de google::LogMessage::Flush() at ??:0
@ 0x7f4a06b0aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4a0ea2a0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f4a0ea338b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4a089d70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4a089cc954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4a098306db start_thread at ??:0
@ 0x7f4a0560761f clone at ??:0
W20260928 04:50:52.849602 15227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:52.849787 15227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:52.849812 15227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:52.850931 15227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:52.850972 15227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:52.850997 15227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:50:52.852510 15227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38331
--tserver_master_addrs=127.0.98.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:52.852738 15227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:52.852972 15227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:52.857860 15261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:52.857887 15264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:52.858032 15227 server_base.cc:1034] running on GCE node
W20260928 04:50:52.857900 15262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:52.858314 15227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:52.858563 15227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:52.859745 15227 hybrid_clock.cc:648] HybridClock initialized: now 1790571052859729 us; error 27 us; skew 500 ppm
Sep 28 04:50:52 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:52.862462 15227 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:50:52.862885 15227 webserver.cc:460] Webserver started at http://127.0.98.1:38331/ using document root <none> and password file <none>
I20260928 04:50:52.863118 15227 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:52.863188 15227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:52.864640 15227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:52.865345 15271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.865595 15227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:52.865701 15227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "317807aa99a7476d954a352fcf1a075e"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "c00963a0ae8d63d570e0224beaa54359"
server_key_iv: "945e1ce0a2f5fdbbeea7dd27e47d3c30"
server_key_version: "encryptionkey@0"
I20260928 04:50:52.866037 15227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:52.879757 15227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:52.882081 15227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:52.882239 15227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:52.882476 15227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:52.882906 15278 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:50:52.884781 15227 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:50:52.884836 15227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:50:52.884893 15227 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:50:52.885562 15227 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:50:52.885615 15227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:50:52.885680 15278 tablet_bootstrap.cc:492] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: Bootstrap starting.
I20260928 04:50:52.892592 15227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36883
I20260928 04:50:52.892655 15385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36883 every 8 connection(s)
I20260928 04:50:52.892936 15227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:50:52.896606 15278 log.cc:824] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:50:52.899056 15278 tablet_bootstrap.cc:492] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:50:52.899360 15278 tablet_bootstrap.cc:492] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: Bootstrap complete.
I20260928 04:50:52.900574 15278 ts_tablet_manager.cc:1397] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260928 04:50:52.901346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 15227
I20260928 04:50:52.901582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44437
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40305
--webserver_port=34837
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:50:52.901871 15278 raft_consensus.cc:348] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.902207 15278 raft_consensus.cc:729] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317807aa99a7476d954a352fcf1a075e, State: Initialized, Role: FOLLOWER
I20260928 04:50:52.902340 15278 consensus_queue.cc:260] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:52.902614 15278 ts_tablet_manager.cc:1428] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260928 04:50:52.961046 15388 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:50:52.893611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:59141)
0928 04:50:52.893780 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.893820 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:52.894233 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:52.960848 (+ 66615us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
W20260928 04:50:52.961340 15386 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40305 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40305: Client connection negotiation failed: client connection to 127.0.98.62:40305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:50:52.961743 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.961594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:34477)
0928 04:50:52.961657 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.961711 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 04:50:52.962121 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:52.962018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:50579)
0928 04:50:52.962067 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:52.962097 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:50:52.962291 15386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:50:52.979956 15391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:52.980149 15391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:52.980170 15391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:52.981244 15391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:52.981280 15391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:52.981304 15391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:50:52.982756 15391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34837
--tserver_master_addrs=127.0.98.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:52.983003 15391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:52.983239 15391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:50:52.987921 15397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:52.987914 15396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:52.988099 15391 server_base.cc:1034] running on GCE node
W20260928 04:50:52.988201 15399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:52.988368 15391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:52.988598 15391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:52.989750 15391 hybrid_clock.cc:648] HybridClock initialized: now 1790571052989750 us; error 40 us; skew 500 ppm
Sep 28 04:50:52 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:52.991966 15391 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:50:52.992359 15391 webserver.cc:460] Webserver started at http://127.0.98.2:34837/ using document root <none> and password file <none>
I20260928 04:50:52.992571 15391 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:52.992638 15391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:52.993917 15391 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:50:52.994486 15406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:52.994650 15391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:50:52.994741 15391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "170cf5c796ff4af9a9e295aab2087cc3"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "ba4058e75d09567d3af0015086aae8ca"
server_key_iv: "32ac7d0ec544c600d19c4d971510ae0a"
server_key_version: "encryptionkey@0"
I20260928 04:50:52.995051 15391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:53.029336 15391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:53.032061 15391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:53.032202 15391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:53.032449 15391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:53.032787 15391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:53.032835 15391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:53.032864 15391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:53.032881 15391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:53.039219 15391 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44437
I20260928 04:50:53.039261 15519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44437 every 8 connection(s)
I20260928 04:50:53.039592 15391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:50:53.040544 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:53.040202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:35195)
0928 04:50:53.040349 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:53.040415 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260928 04:50:53.040922 15520 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40305 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40305: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
I20260928 04:50:53.041245 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:53.041118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:46257)
0928 04:50:53.041159 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:53.041220 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 04:50:53.041559 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:53.041466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:50199)
0928 04:50:53.041510 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:53.041545 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:50:53.041668 15520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:50:53.046646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 15391
I20260928 04:50:53.046895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43703
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40305
--webserver_port=41463
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:50:53.127205 15523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:50:53.127372 15523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:50:53.127393 15523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:50:53.128545 15523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:50:53.128585 15523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:50:53.128615 15523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:50:53.129981 15523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41463
--tserver_master_addrs=127.0.98.62:40305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:50:53.130223 15523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:50:53.130420 15523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:50:53.135108 15523 server_base.cc:1034] running on GCE node
W20260928 04:50:53.135074 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:53.135114 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:50:53.135078 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:50:53.135390 15523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:50:53.135641 15523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:50:53.136775 15523 hybrid_clock.cc:648] HybridClock initialized: now 1790571053136756 us; error 33 us; skew 500 ppm
Sep 28 04:50:53 dist-test-slave-jccs krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:50:53.139050 15523 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:50:53.139480 15523 webserver.cc:460] Webserver started at http://127.0.98.3:41463/ using document root <none> and password file <none>
I20260928 04:50:53.139711 15523 fs_manager.cc:359] Metadata directory not provided
I20260928 04:50:53.139778 15523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:50:53.140977 15523 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:50:53.141577 15538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:53.141736 15523 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:53.141827 15523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd"
format_stamp: "Formatted at 2026-09-28 04:50:51 on dist-test-slave-jccs"
server_key: "f405870892a3503356aa30568ed8068c"
server_key_iv: "46dc594d2eb06ca92f66063b59328112"
server_key_version: "encryptionkey@0"
I20260928 04:50:53.142162 15523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:50:53.151539 15523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:50:53.153355 15523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:50:53.153504 15523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:50:53.153719 15523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:50:53.153991 15523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:50:53.154043 15523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:53.154076 15523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:50:53.154116 15523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:50:53.159469 15523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43703
I20260928 04:50:53.159516 15651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43703 every 8 connection(s)
I20260928 04:50:53.159817 15523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:50:53.160723 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:53.160398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:41833)
0928 04:50:53.160536 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:53.160603 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
W20260928 04:50:53.161072 15652 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40305 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40305: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
I20260928 04:50:53.161324 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:53.161262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:49937)
0928 04:50:53.161286 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:53.161308 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
I20260928 04:50:53.161561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 15523
I20260928 04:50:53.161621 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:53.161520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:38243)
0928 04:50:53.161566 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:53.161600 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 04:50:53.161727 15652 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:50:53.962805 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:53.962596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:32907)
0928 04:50:53.962711 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:53.962745 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260928 04:50:54.042117 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:54.041930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:37051)
0928 04:50:54.042040 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:54.042068 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20260928 04:50:54.162231 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:54.162041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:33733)
0928 04:50:54.162156 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:54.162189 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260928 04:50:54.473484 15659 raft_consensus.cc:482] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:50:54.473610 15659 raft_consensus.cc:504] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:54.473955 15659 leader_election.cc:288] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437), 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703)
Sep 28 04:50:54 dist-test-slave-jccs krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 28 04:50:54 dist-test-slave-jccs krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 04:50:54.483769 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:54.474241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59221 (local address 127.0.98.3:43703)
0928 04:50:54.474378 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:54.474385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:54.474407 (+ 22us) server_negotiation.cc:407] Connection header received
0928 04:50:54.474800 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:54.474802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:54.474839 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:54.475003 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:54.475657 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:54.476461 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:54.477558 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:54.477927 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:54.478062 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:54.480259 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:54.480263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:54.480263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:54.480272 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:54.482024 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:54.482535 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:54.482538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:54.482538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:54.482640 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:54.482903 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:54.482904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:54.482904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:54.483121 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:54.483234 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:54.483529 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 04:50:54.483641 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260928 04:50:54.483770 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:54.474078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44437 (local address 127.0.98.1:48343)
0928 04:50:54.474194 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:54.474204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:54.474258 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:54.475030 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:54.475032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:54.475375 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:54.475551 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:54.475556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:54.476542 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:54.476543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:54.477454 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:54.477457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:54.478040 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:54.478041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:54.478141 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:54.478740 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:54.478745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:54.480532 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:54.482269 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:54.482270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:54.482270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:54.482472 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:54.482728 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:54.482729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:54.482729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:54.482837 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:54.483203 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:54.483209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:54.483346 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 04:50:54.483467 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 04:50:54.483651 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":71,"thread_start_us":64,"threads_started":1}
I20260928 04:50:54.483793 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:54.474171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48343 (local address 127.0.98.2:44437)
0928 04:50:54.474316 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 04:50:54.474328 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:50:54.474361 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:50:54.474800 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:50:54.474802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:50:54.474839 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:50:54.475005 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:50:54.475690 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:54.476448 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:54.477561 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:50:54.477935 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:50:54.478068 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:54.480623 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:50:54.480626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:50:54.480626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:50:54.480635 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:50:54.482172 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:54.482567 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:54.482569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:54.482569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:54.482645 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:50:54.482903 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:50:54.482904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:50:54.482905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:50:54.483121 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:50:54.483209 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:50:54.483529 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 04:50:54.483644 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":45,"threads_started":1}
I20260928 04:50:54.483883 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:54.474108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43703 (local address 127.0.98.1:59221)
0928 04:50:54.474206 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:54.474214 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:50:54.474264 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:50:54.475029 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:50:54.475032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:50:54.475374 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:50:54.475551 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:54.475556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:54.476546 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:54.476547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:54.477453 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:50:54.477455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:50:54.478039 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:50:54.478040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:50:54.478143 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:50:54.478670 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:50:54.478679 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:50:54.480164 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:50:54.482105 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:54.482111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:54.482113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:54.482459 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:54.482709 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:50:54.482710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:50:54.482711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:50:54.482836 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:50:54.483219 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:50:54.483220 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:50:54.483345 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 04:50:54.483467 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 04:50:54.483748 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":125,"thread_start_us":54,"threads_started":1}
W20260928 04:50:54.485116 15273 leader_election.cc:341] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437): Not found: Tablet not found: 8db864a6f96d45028e98c26e6ef14717
W20260928 04:50:54.485249 15272 leader_election.cc:341] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703): Not found: Tablet not found: 8db864a6f96d45028e98c26e6ef14717
I20260928 04:50:54.485299 15272 leader_election.cc:302] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 317807aa99a7476d954a352fcf1a075e; no voters: 170cf5c796ff4af9a9e295aab2087cc3, 8bcf7ba63ea24d0fa23d1a948b79f6dd
I20260928 04:50:54.485417 15659 raft_consensus.cc:2726] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:50:54.963313 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:54.963181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:36757)
0928 04:50:54.963244 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:54.963277 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 04:50:55.042689 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:55.042504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:33591)
0928 04:50:55.042613 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:55.042654 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260928 04:50:55.162823 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:55.162642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:58917)
0928 04:50:55.162753 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:55.162786 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260928 04:50:55.963972 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:55.963739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:36201)
0928 04:50:55.963894 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:55.963928 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260928 04:50:56.043318 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:56.043120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:52803)
0928 04:50:56.043244 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:56.043279 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260928 04:50:56.163564 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:56.163334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:50017)
0928 04:50:56.163478 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:56.163511 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260928 04:50:56.660686 15669 raft_consensus.cc:482] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:50:56.660779 15669 raft_consensus.cc:504] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:56.660929 15669 leader_election.cc:288] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437), 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703)
W20260928 04:50:56.661327 15273 leader_election.cc:341] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437): Not found: Tablet not found: 8db864a6f96d45028e98c26e6ef14717
W20260928 04:50:56.661420 15272 leader_election.cc:341] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703): Not found: Tablet not found: 8db864a6f96d45028e98c26e6ef14717
I20260928 04:50:56.661495 15272 leader_election.cc:302] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 317807aa99a7476d954a352fcf1a075e; no voters: 170cf5c796ff4af9a9e295aab2087cc3, 8bcf7ba63ea24d0fa23d1a948b79f6dd
I20260928 04:50:56.661580 15669 raft_consensus.cc:2726] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:50:56.964610 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:56.964401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:55441)
0928 04:50:56.964530 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:56.964564 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260928 04:50:57.044006 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:57.043805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:38617)
0928 04:50:57.043924 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:57.043963 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260928 04:50:57.164182 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:57.163993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:52823)
0928 04:50:57.164096 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:57.164147 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260928 04:50:57.965391 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:57.965053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:54745)
0928 04:50:57.965314 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:57.965349 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":69,"threads_started":1}
I20260928 04:50:58.044708 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:58.044483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:53001)
0928 04:50:58.044587 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:58.044664 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
I20260928 04:50:58.164799 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:58.164608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:51027)
0928 04:50:58.164721 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:58.164761 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 04:50:58.820096 15676 raft_consensus.cc:482] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:50:58.820194 15676 raft_consensus.cc:504] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:50:58.820360 15676 leader_election.cc:288] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437), 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703)
W20260928 04:50:58.820672 15273 leader_election.cc:341] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437): Not found: Tablet not found: 8db864a6f96d45028e98c26e6ef14717
W20260928 04:50:58.820762 15272 leader_election.cc:341] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703): Not found: Tablet not found: 8db864a6f96d45028e98c26e6ef14717
I20260928 04:50:58.820820 15272 leader_election.cc:302] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 317807aa99a7476d954a352fcf1a075e; no voters: 170cf5c796ff4af9a9e295aab2087cc3, 8bcf7ba63ea24d0fa23d1a948b79f6dd
I20260928 04:50:58.820909 15676 raft_consensus.cc:2726] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:50:58.966043 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:58.965840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:51469)
0928 04:50:58.965961 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:58.965999 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260928 04:50:59.045389 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:59.045154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:46915)
0928 04:50:59.045311 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:59.045349 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260928 04:50:59.165531 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:59.165347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:54603)
0928 04:50:59.165453 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:59.165493 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260928 04:50:59.966822 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:50:59.966611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:52005)
0928 04:50:59.966743 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 04:50:59.966776 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260928 04:51:00.045982 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:00.045793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:33601)
0928 04:51:00.045905 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:00.045939 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20260928 04:51:00.166257 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:00.166065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:38397)
0928 04:51:00.166189 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:00.166222 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260928 04:51:00.967418 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:00.967230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:57555)
0928 04:51:00.967343 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:00.967373 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260928 04:51:01.046638 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:01.046437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:40773)
0928 04:51:01.046550 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:01.046585 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260928 04:51:01.166867 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:01.166674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:43357)
0928 04:51:01.166784 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:01.166826 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260928 04:51:01.968155 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:01.967879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:48179)
0928 04:51:01.968047 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:01.968109 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260928 04:51:02.047407 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:02.047182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:53439)
0928 04:51:02.047309 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:02.047348 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":53,"threads_started":1}
I20260928 04:51:02.167587 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:02.167370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:54613)
0928 04:51:02.167500 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:02.167537 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260928 04:51:02.874694 15689 raft_consensus.cc:482] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:51:02.874809 15689 raft_consensus.cc:504] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170cf5c796ff4af9a9e295aab2087cc3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44437 } } peers { permanent_uuid: "8bcf7ba63ea24d0fa23d1a948b79f6dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43703 } } peers { permanent_uuid: "317807aa99a7476d954a352fcf1a075e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36883 } }
I20260928 04:51:02.875000 15689 leader_election.cc:288] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437), 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703)
W20260928 04:51:02.875447 15272 leader_election.cc:341] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8bcf7ba63ea24d0fa23d1a948b79f6dd (127.0.98.3:43703): Not found: Tablet not found: 8db864a6f96d45028e98c26e6ef14717
W20260928 04:51:02.875519 15273 leader_election.cc:341] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 170cf5c796ff4af9a9e295aab2087cc3 (127.0.98.2:44437): Not found: Tablet not found: 8db864a6f96d45028e98c26e6ef14717
I20260928 04:51:02.875584 15273 leader_election.cc:302] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 317807aa99a7476d954a352fcf1a075e; no voters: 170cf5c796ff4af9a9e295aab2087cc3, 8bcf7ba63ea24d0fa23d1a948b79f6dd
I20260928 04:51:02.875662 15689 raft_consensus.cc:2726] T 8db864a6f96d45028e98c26e6ef14717 P 317807aa99a7476d954a352fcf1a075e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 04:51:02.968945 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:02.968658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:41875)
0928 04:51:02.968838 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:02.968893 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":56,"threads_started":1}
I20260928 04:51:03.048080 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:03.047873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:44573)
0928 04:51:03.047995 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:03.048027 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260928 04:51:03.168380 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:03.168151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:46385)
0928 04:51:03.168274 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:03.168310 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260928 04:51:03.969653 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:03.969431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:43405)
0928 04:51:03.969559 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:03.969603 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260928 04:51:04.048688 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:04.048478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:56811)
0928 04:51:04.048608 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:04.048642 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260928 04:51:04.169234 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:04.168881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:45785)
0928 04:51:04.169065 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:04.169111 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260928 04:51:04.970319 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:04.970112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:44013)
0928 04:51:04.970227 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:04.970263 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260928 04:51:05.049515 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:05.049313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:48335)
0928 04:51:05.049431 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:05.049471 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":53,"threads_started":1}
I20260928 04:51:05.170028 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:05.169743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:38687)
0928 04:51:05.169904 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:05.169957 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260928 04:51:05.970938 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:05.970728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:39471)
0928 04:51:05.970860 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:05.970893 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260928 04:51:06.050175 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:06.049953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:54629)
0928 04:51:06.050096 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:06.050132 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260928 04:51:06.170710 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:06.170496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:55963)
0928 04:51:06.170626 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:06.170663 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260928 04:51:06.971617 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:06.971393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:45345)
0928 04:51:06.971538 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:06.971569 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260928 04:51:07.050818 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:07.050609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:57125)
0928 04:51:07.050732 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:07.050771 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260928 04:51:07.171301 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:07.171113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:57067)
0928 04:51:07.171226 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:07.171259 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 04:51:07.972352 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:07.972150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.1:60443)
0928 04:51:07.972269 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:07.972307 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260928 04:51:08.051503 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:08.051292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.2:57211)
0928 04:51:08.051428 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:08.051461 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
I20260928 04:51:08.171002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 15227
I20260928 04:51:08.172008 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:08.171811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40305 (local address 127.0.98.3:56149)
0928 04:51:08.171934 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:08.171967 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260928 04:51:08.175977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 15391
I20260928 04:51:08.180138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 15523
2026-09-28T04:51:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18451 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:51:08 dist-test-slave-jccs krb5kdc[15710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:51:08 dist-test-slave-jccs krb5kdc[15710](info): set up 2 sockets
Sep 28 04:51:08 dist-test-slave-jccs krb5kdc[15710](info): commencing operation
krb5kdc: starting...
W20260928 04:51:09.185352 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:51:09 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:51:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:51:09.295894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33349
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:51:09.373984 15723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:09.374186 15723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:09.374207 15723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:09.375293 15723 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 04:51:09.375329 15723 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 04:51:09.375341 15723 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 04:51:09.375353 15723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:51:09.375401 15723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:09.375418 15723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:09.375440 15723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:51:09.375466 15723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:51:09.376874 15723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:09.377048 15723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:09.377293 15723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:51:09.382302 15729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:09.382311 15728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:09.382334 15723 server_base.cc:1034] running on GCE node
W20260928 04:51:09.382311 15731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:09.382795 15723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:09.383085 15723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:09.384238 15723 hybrid_clock.cc:648] HybridClock initialized: now 1790571069384226 us; error 27 us; skew 500 ppm
Sep 28 04:51:09 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:09.386806 15723 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:51:09.387285 15723 webserver.cc:460] Webserver started at http://127.0.98.62:40877/ using document root <none> and password file <none>
I20260928 04:51:09.387506 15723 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:09.387574 15723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:09.387720 15723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:51:09.388731 15723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "1cbc006c0baca004b0a0085d91203d7a"
server_key_iv: "ba1b861d7131e7ee197b35b2c20fd273"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.389128 15723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "1cbc006c0baca004b0a0085d91203d7a"
server_key_iv: "ba1b861d7131e7ee197b35b2c20fd273"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.390455 15723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:51:09.391162 15738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:09.391317 15723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:51:09.391415 15723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "1cbc006c0baca004b0a0085d91203d7a"
server_key_iv: "ba1b861d7131e7ee197b35b2c20fd273"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.391513 15723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:09.406529 15723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:09.408188 15723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:09.408411 15723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:09.412381 15723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33349
I20260928 04:51:09.412425 15790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33349 every 8 connection(s)
I20260928 04:51:09.412772 15723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:51:09.413625 15791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:51:09.415625 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Bootstrap starting.
I20260928 04:51:09.416126 15791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Neither blocks nor log segments found. Creating new log.
I20260928 04:51:09.416354 15791 log.cc:824] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:09.416922 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: No bootstrap required, opened a new log
I20260928 04:51:09.418447 15791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:09.418569 15791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:51:09.418589 15791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ed38cc827b4cf2a8b17dbd9f4eb1af, State: Initialized, Role: FOLLOWER
I20260928 04:51:09.418717 15791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:09.418792 15791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:51:09.418812 15791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:51:09.418838 15791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:51:09.419283 15791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:09.419364 15791 leader_election.cc:302] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29ed38cc827b4cf2a8b17dbd9f4eb1af; no voters:
I20260928 04:51:09.419479 15791 leader_election.cc:288] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:51:09.419569 15794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:51:09.419720 15794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 1 LEADER]: Becoming Leader. State: Replica: 29ed38cc827b4cf2a8b17dbd9f4eb1af, State: Running, Role: LEADER
I20260928 04:51:09.419734 15791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:51:09.419801 15794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:09.420186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 15723
I20260928 04:51:09.420289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:51:09.420377 15795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } } }
I20260928 04:51:09.420473 15795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: This master's current role is: LEADER
I20260928 04:51:09.420569 392 external_mini_cluster.cc:1428] Setting key 36962a4621868a2e9a8a2277bb0a1750
I20260928 04:51:09.420727 15803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:51:09.421281 15803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:51:09.422303 15796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ed38cc827b4cf2a8b17dbd9f4eb1af. Latest consensus state: current_term: 1 leader_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } } }
I20260928 04:51:09.422569 15796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: This master's current role is: LEADER
I20260928 04:51:09.423014 15803 catalog_manager.cc:1347] Generated new cluster ID: f1bbe07742b24d99bc958dc1583ad158
I20260928 04:51:09.423074 15803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:51:09 dist-test-slave-jccs krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:51:09.432219 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.421161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.0.98.62:33349)
0928 04:51:09.421349 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.421358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.421393 (+ 35us) server_negotiation.cc:407] Connection header received
0928 04:51:09.421682 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.421689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.421742 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.421871 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:09.422865 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.424275 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.425327 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.425954 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.426077 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.428796 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:09.428799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:09.428800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:09.428810 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:09.430666 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:09.431048 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:09.431051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:09.431051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:09.431141 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:09.431343 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:09.431344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:09.431344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:09.431609 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:09.431727 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.431856 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.431974 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260928 04:51:09.435196 15803 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:51:09.435642 15803 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:51:09.441605 15803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Generated new TSK 0
I20260928 04:51:09.441757 15803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:51:09.467859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33349
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:51:09.547287 15817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:09.547489 15817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:09.547513 15817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:09.548676 15817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:09.548717 15817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:09.548769 15817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:51:09.550312 15817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:09.550562 15817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:09.550807 15817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:51:09.555791 15823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:09.555841 15822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:09.555814 15825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:09.555923 15817 server_base.cc:1034] running on GCE node
I20260928 04:51:09.556164 15817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:09.556387 15817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:09.557546 15817 hybrid_clock.cc:648] HybridClock initialized: now 1790571069557533 us; error 24 us; skew 500 ppm
Sep 28 04:51:09 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:09.559823 15817 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:51:09.560263 15817 webserver.cc:460] Webserver started at http://127.0.98.1:38959/ using document root <none> and password file <none>
I20260928 04:51:09.560482 15817 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:09.560547 15817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:09.560709 15817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:51:09.561766 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "905806324bdb4c0ca488ff07d4ced7ea"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "23919486946cd82e196e65f8cee1d02c"
server_key_iv: "8e6a3a9ea9a3382772afb19ee47d176a"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.562180 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "905806324bdb4c0ca488ff07d4ced7ea"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "23919486946cd82e196e65f8cee1d02c"
server_key_iv: "8e6a3a9ea9a3382772afb19ee47d176a"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.563439 15817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:51:09.564116 15832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:09.564280 15817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:51:09.564388 15817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "905806324bdb4c0ca488ff07d4ced7ea"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "23919486946cd82e196e65f8cee1d02c"
server_key_iv: "8e6a3a9ea9a3382772afb19ee47d176a"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.564456 15817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:09.589133 15817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:09.591007 15817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:09.591167 15817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:09.591373 15817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:51:09.591671 15817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:51:09.591728 15817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:09.591769 15817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:51:09.591795 15817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:09.597553 15817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34037
I20260928 04:51:09.597599 15945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34037 every 8 connection(s)
I20260928 04:51:09.597892 15817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:51:09.602504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 15817
I20260928 04:51:09.602615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:51:09.602838 392 external_mini_cluster.cc:1428] Setting key 09bbbeacbe46f20433444fd2e4cbfa06
Sep 28 04:51:09 dist-test-slave-jccs krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:51:09.607023 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.598549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50657 (local address 127.0.98.62:33349)
0928 04:51:09.598623 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.598626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.598799 (+ 173us) server_negotiation.cc:407] Connection header received
0928 04:51:09.599264 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.599265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.599279 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.599326 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:09.600201 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.600691 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.601669 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.601866 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.601919 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.604115 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:09.604118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:09.604118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:09.604127 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:09.605618 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:09.606068 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:09.606069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:09.606070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:09.606129 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:09.606365 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:09.606366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:09.606366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:09.606508 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:09.606587 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.606811 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.606903 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:51:09.607052 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.598514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:50657)
0928 04:51:09.598686 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:09.598719 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:09.599183 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:09.599382 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:09.599385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:09.599944 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:51:09.600126 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.600130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.600765 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.600765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.601595 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.601597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.601937 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.601938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.602031 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.602482 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:51:09.602494 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:51:09.603998 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:51:09.605698 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:09.605705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:09.605707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:09.605988 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:09.606205 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:09.606205 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:09.606206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:09.606299 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:09.606581 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:51:09.606585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:51:09.606685 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 04:51:09.606763 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:51:09.606869 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260928 04:51:09.607599 15946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33349
I20260928 04:51:09.607712 15946 heartbeater.cc:464] Registering TS with master...
I20260928 04:51:09.607872 15946 heartbeater.cc:511] Master 127.0.98.62:33349 requested a full tablet report, sending...
I20260928 04:51:09.608465 15755 ts_manager.cc:194] Registered new tserver with Master: 905806324bdb4c0ca488ff07d4ced7ea (127.0.98.1:34037)
I20260928 04:51:09.609413 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50657
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:51:09.627331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33349
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:51:09.705992 15951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:09.706187 15951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:09.706209 15951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:09.707267 15951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:09.707305 15951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:09.707331 15951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:51:09.708678 15951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:09.708912 15951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:09.709146 15951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:51:09.713824 15956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:09.713852 15957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:09.713824 15959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:09.714064 15951 server_base.cc:1034] running on GCE node
I20260928 04:51:09.714293 15951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:09.714515 15951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:09.715651 15951 hybrid_clock.cc:648] HybridClock initialized: now 1790571069715632 us; error 35 us; skew 500 ppm
Sep 28 04:51:09 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:09.717986 15951 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:51:09.718396 15951 webserver.cc:460] Webserver started at http://127.0.98.2:46287/ using document root <none> and password file <none>
I20260928 04:51:09.718621 15951 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:09.718690 15951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:09.718837 15951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:51:09.719834 15951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "8805768fa4594f3fb3e5620d1a5641a6"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "77be6f12a0354ec8682831656da3f083"
server_key_iv: "134a9c3105daef761047429557c0df98"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.720206 15951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8805768fa4594f3fb3e5620d1a5641a6"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "77be6f12a0354ec8682831656da3f083"
server_key_iv: "134a9c3105daef761047429557c0df98"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.721491 15951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:51:09.722280 15966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:09.722446 15951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:51:09.722543 15951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "8805768fa4594f3fb3e5620d1a5641a6"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "77be6f12a0354ec8682831656da3f083"
server_key_iv: "134a9c3105daef761047429557c0df98"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.722621 15951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:09.742920 15951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:09.744694 15951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:09.744848 15951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:09.745059 15951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:51:09.745415 15951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:51:09.745463 15951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:09.745494 15951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:51:09.745519 15951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:09.751129 15951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39971
I20260928 04:51:09.751168 16079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39971 every 8 connection(s)
I20260928 04:51:09.751475 15951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:51:09 dist-test-slave-jccs krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:51:09.759662 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.752157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48263 (local address 127.0.98.62:33349)
0928 04:51:09.752220 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.752222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.752397 (+ 175us) server_negotiation.cc:407] Connection header received
0928 04:51:09.752870 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.752871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.752882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.752929 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:09.753815 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.754297 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.755144 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.755306 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.755372 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.757375 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:09.757377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:09.757377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:09.757382 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:09.758490 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:09.758837 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:09.758838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:09.758838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:09.758883 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:09.759119 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:09.759121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:09.759121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:09.759235 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:09.759301 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.759511 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.759598 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:51:09.759776 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.752143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:48263)
0928 04:51:09.752298 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:09.752331 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:09.752785 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:09.752985 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:09.752988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:09.753553 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:51:09.753742 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.753747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.754367 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.754368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.755072 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.755074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.755406 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.755406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.755496 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.755985 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:51:09.755992 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:51:09.757291 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:51:09.758564 (+ 1273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:09.758569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:09.758571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:09.758776 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:09.758937 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:09.758938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:09.758938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:09.759026 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:09.759300 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:51:09.759304 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:51:09.759404 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 04:51:09.759480 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 04:51:09.759590 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260928 04:51:09.760291 16080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33349
I20260928 04:51:09.760394 16080 heartbeater.cc:464] Registering TS with master...
I20260928 04:51:09.760605 16080 heartbeater.cc:511] Master 127.0.98.62:33349 requested a full tablet report, sending...
I20260928 04:51:09.760980 15755 ts_manager.cc:194] Registered new tserver with Master: 8805768fa4594f3fb3e5620d1a5641a6 (127.0.98.2:39971)
I20260928 04:51:09.761421 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48263
I20260928 04:51:09.761654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 15951
I20260928 04:51:09.761777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:51:09.761996 392 external_mini_cluster.cc:1428] Setting key 5d9445388a1f64e242021b4f4789daa9
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:51:09.785571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33349
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:51:09.864879 16085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:09.865077 16085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:09.865098 16085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:09.866290 16085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:09.866333 16085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:09.866358 16085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:51:09.867781 16085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:09.867960 16085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:09.868135 16085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:51:09.872895 16091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:09.872901 16090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:09.872895 16093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:09.873131 16085 server_base.cc:1034] running on GCE node
I20260928 04:51:09.873344 16085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:09.873584 16085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:09.874722 16085 hybrid_clock.cc:648] HybridClock initialized: now 1790571069874707 us; error 35 us; skew 500 ppm
Sep 28 04:51:09 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:09.877079 16085 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:51:09.877535 16085 webserver.cc:460] Webserver started at http://127.0.98.3:45375/ using document root <none> and password file <none>
I20260928 04:51:09.877702 16085 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:09.877743 16085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:09.877823 16085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:51:09.878731 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "48a41c7e18d14dc0b4c5994622fe1214"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "91a4e36d381f49f6a595f9d1a19c82e1"
server_key_iv: "0ccc24ffd244742426b2a726ce3440cd"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.879007 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "48a41c7e18d14dc0b4c5994622fe1214"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "91a4e36d381f49f6a595f9d1a19c82e1"
server_key_iv: "0ccc24ffd244742426b2a726ce3440cd"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.880175 16085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:51:09.880820 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:09.880998 16085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:51:09.881063 16085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "48a41c7e18d14dc0b4c5994622fe1214"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "91a4e36d381f49f6a595f9d1a19c82e1"
server_key_iv: "0ccc24ffd244742426b2a726ce3440cd"
server_key_version: "encryptionkey@0"
I20260928 04:51:09.881147 16085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:09.899672 16085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:09.901994 16085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:09.902153 16085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:09.902403 16085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:51:09.902709 16085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:51:09.902765 16085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:09.902805 16085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:51:09.902833 16085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:09.908807 16085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36545
I20260928 04:51:09.908844 16213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36545 every 8 connection(s)
I20260928 04:51:09.909236 16085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:51:09.909757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 16085
I20260928 04:51:09.909873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:51:09.910104 392 external_mini_cluster.cc:1428] Setting key bb8ec947123563dc8fbfd3fb8bb6a8cb
Sep 28 04:51:09 dist-test-slave-jccs krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:51:09.917873 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.909949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44915 (local address 127.0.98.62:33349)
0928 04:51:09.909999 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.910003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.910236 (+ 233us) server_negotiation.cc:407] Connection header received
0928 04:51:09.910936 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.910937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.910948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.911008 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:09.911911 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.912403 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.913438 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.913607 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.913659 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.915497 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:09.915498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:09.915498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:09.915504 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:09.916802 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:09.917119 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:09.917120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:09.917120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:09.917165 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:09.917414 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:09.917414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:09.917414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:09.917507 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:09.917553 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.917761 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.917816 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:51:09.918004 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.909945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:44915)
0928 04:51:09.910106 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:09.910152 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:09.910814 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:09.911085 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:09.911095 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:09.911658 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:51:09.911835 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.911839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.912478 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.912479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.913357 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.913359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.913675 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.913675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.913765 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.914158 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:51:09.914165 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:51:09.915418 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:51:09.916870 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:09.916875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:09.916877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:09.917060 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:09.917272 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:09.917273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:09.917273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:09.917361 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:09.917558 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:51:09.917562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:51:09.917662 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 04:51:09.917736 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 04:51:09.917835 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260928 04:51:09.918453 16214 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33349
I20260928 04:51:09.918522 16214 heartbeater.cc:464] Registering TS with master...
I20260928 04:51:09.918648 16214 heartbeater.cc:511] Master 127.0.98.62:33349 requested a full tablet report, sending...
I20260928 04:51:09.918967 15755 ts_manager.cc:194] Registered new tserver with Master: 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545)
I20260928 04:51:09.919335 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44915
I20260928 04:51:09.921790 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:51:09.929045 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.923218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.0.98.62:33349)
0928 04:51:09.923292 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.923294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.923365 (+ 71us) server_negotiation.cc:407] Connection header received
0928 04:51:09.923440 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.923441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.923450 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.923504 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:09.924058 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.924520 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.925385 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.925569 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.925651 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.926796 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:09.926798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:09.926798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:09.926805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:09.927903 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:09.928243 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:09.928244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:09.928244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:09.928300 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:09.928497 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:09.928498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:09.928498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:09.928633 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:09.928733 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.928883 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.928954 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 04:51:09.930442 15755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:51:09.931938 15755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:51:09.937935 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.934064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.0.1:44754)
0928 04:51:09.934295 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:09.934335 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:09.934402 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:09.934737 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:09.934740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:09.934742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:51:09.934801 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.934803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.936108 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.936109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.936814 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.936816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.937616 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.937617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.937709 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.937721 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 04:51:09.937818 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 04:51:09.937867 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":52,"threads_started":1}
I20260928 04:51:09.937947 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.934401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.0.98.1:34037)
0928 04:51:09.934570 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.934573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.934582 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:51:09.934617 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.934619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.934635 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.934728 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:51:09.934880 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.936007 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.936905 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.937522 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.937606 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.937660 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.937836 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.937874 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260928 04:51:09.938189 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.934226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.0.98.3:36545)
0928 04:51:09.934404 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.934409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.934577 (+ 168us) server_negotiation.cc:407] Connection header received
0928 04:51:09.934665 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.934667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.934680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.934754 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:51:09.934941 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.935746 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.937053 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.937762 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.937808 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.937853 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.938067 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.938103 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260928 04:51:09.938266 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.934090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39971 (local address 127.0.0.1:58150)
0928 04:51:09.934571 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:09.934580 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:09.934624 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:09.934835 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:09.934835 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:09.934837 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:51:09.934868 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.934870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.936086 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.936087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.937117 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.937119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.938044 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.938045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.938094 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.938097 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:51:09.938148 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:51:09.938184 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":49,"threads_started":1}
I20260928 04:51:09.938295 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.934300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.0.98.2:39971)
0928 04:51:09.934486 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.934490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.934677 (+ 187us) server_negotiation.cc:407] Connection header received
0928 04:51:09.934739 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.934741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.934756 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.934825 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:51:09.934922 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.935966 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.937234 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.937910 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.937967 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.938016 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.938166 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.938208 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260928 04:51:09.938438 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.934064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.0.1:47914)
0928 04:51:09.934520 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:09.934530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:09.934580 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:09.934817 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:09.934818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:09.934820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:51:09.934868 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.934871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.935836 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.935836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.936961 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.936964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.937916 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.937917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.937979 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.937982 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:51:09.938308 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 04:51:09.938375 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":53,"threads_started":1}
I20260928 04:51:09.938938 16014 tablet_service.cc:1467] Processing CreateTablet for tablet 6b57cff9d0504fdcaf67730ddbbf7376 (DEFAULT_TABLE table=test-table [id=1dcfb66266a54f8aada42a34a63a875c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:51:09.938989 16148 tablet_service.cc:1467] Processing CreateTablet for tablet 6b57cff9d0504fdcaf67730ddbbf7376 (DEFAULT_TABLE table=test-table [id=1dcfb66266a54f8aada42a34a63a875c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:51:09.939167 15880 tablet_service.cc:1467] Processing CreateTablet for tablet 6b57cff9d0504fdcaf67730ddbbf7376 (DEFAULT_TABLE table=test-table [id=1dcfb66266a54f8aada42a34a63a875c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:51:09.939277 16148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b57cff9d0504fdcaf67730ddbbf7376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:51:09.939277 16014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b57cff9d0504fdcaf67730ddbbf7376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:51:09.939488 15880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b57cff9d0504fdcaf67730ddbbf7376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:51:09.941885 16228 tablet_bootstrap.cc:492] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Bootstrap starting.
I20260928 04:51:09.942534 16230 tablet_bootstrap.cc:492] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: Bootstrap starting.
I20260928 04:51:09.942744 16228 tablet_bootstrap.cc:654] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Neither blocks nor log segments found. Creating new log.
I20260928 04:51:09.943113 16230 tablet_bootstrap.cc:654] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: Neither blocks nor log segments found. Creating new log.
I20260928 04:51:09.943274 16228 log.cc:824] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:09.943671 16230 log.cc:824] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:09.943954 16229 tablet_bootstrap.cc:492] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: Bootstrap starting.
I20260928 04:51:09.944063 16228 tablet_bootstrap.cc:492] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: No bootstrap required, opened a new log
I20260928 04:51:09.944166 16228 ts_tablet_manager.cc:1397] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:51:09.944389 16230 tablet_bootstrap.cc:492] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: No bootstrap required, opened a new log
I20260928 04:51:09.944456 16230 ts_tablet_manager.cc:1397] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:51:09.944602 16229 tablet_bootstrap.cc:654] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: Neither blocks nor log segments found. Creating new log.
I20260928 04:51:09.945055 16229 log.cc:824] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:09.945801 16229 tablet_bootstrap.cc:492] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: No bootstrap required, opened a new log
I20260928 04:51:09.945887 16229 ts_tablet_manager.cc:1397] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:51:09.946174 16228 raft_consensus.cc:348] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:09.946365 16228 raft_consensus.cc:374] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:51:09.946399 16228 raft_consensus.cc:729] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905806324bdb4c0ca488ff07d4ced7ea, State: Initialized, Role: FOLLOWER
I20260928 04:51:09.946532 16228 consensus_queue.cc:260] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:09.946552 16230 raft_consensus.cc:348] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:09.946712 16230 raft_consensus.cc:374] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:51:09.946758 16230 raft_consensus.cc:729] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a41c7e18d14dc0b4c5994622fe1214, State: Initialized, Role: FOLLOWER
I20260928 04:51:09.946811 16228 ts_tablet_manager.cc:1428] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:51:09.946877 16230 consensus_queue.cc:260] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:09.946993 15946 heartbeater.cc:503] Master 127.0.98.62:33349 was elected leader, sending a full tablet report...
I20260928 04:51:09.947149 16214 heartbeater.cc:503] Master 127.0.98.62:33349 was elected leader, sending a full tablet report...
I20260928 04:51:09.947155 16230 ts_tablet_manager.cc:1428] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:51:09.947695 16229 raft_consensus.cc:348] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:09.947865 16229 raft_consensus.cc:374] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:51:09.947906 16229 raft_consensus.cc:729] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8805768fa4594f3fb3e5620d1a5641a6, State: Initialized, Role: FOLLOWER
I20260928 04:51:09.948030 16229 consensus_queue.cc:260] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:09.948298 16229 ts_tablet_manager.cc:1428] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:51:09.948312 16080 heartbeater.cc:503] Master 127.0.98.62:33349 was elected leader, sending a full tablet report...
I20260928 04:51:09.967286 16236 raft_consensus.cc:482] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:51:09.967422 16236 raft_consensus.cc:504] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:09.967756 16236 leader_election.cc:288] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545), 905806324bdb4c0ca488ff07d4ced7ea (127.0.98.1:34037)
I20260928 04:51:09.971460 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.967999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32849 (local address 127.0.98.1:34037)
0928 04:51:09.968037 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.968039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.968047 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:51:09.968072 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.968077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.968137 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.968206 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:51:09.968394 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.969096 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.970546 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.971096 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.971173 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.971182 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.971358 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.971394 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:51:09.971460 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.967914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46113 (local address 127.0.98.3:36545)
0928 04:51:09.967951 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.967953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.968116 (+ 163us) server_negotiation.cc:407] Connection header received
0928 04:51:09.968144 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.968145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.968156 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.968209 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:51:09.968395 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.969097 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.970547 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.971096 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.971173 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.971182 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.971359 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.971394 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:51:09.971495 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.967887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.98.2:32849)
0928 04:51:09.967936 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:09.967948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:09.967987 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:09.968232 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:09.968233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:09.968234 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:51:09.968324 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.968326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.969203 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.969204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.970449 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.970452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.971184 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.971185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.971278 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.971284 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:51:09.971340 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:51:09.971389 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":113}
I20260928 04:51:09.971503 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.967887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.98.2:46113)
0928 04:51:09.968038 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:09.968047 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:09.968099 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:09.968222 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:09.968222 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:09.968224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:51:09.968324 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.968326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.969238 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.969238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.970449 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:09.970452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.971184 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.971185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:09.971278 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.971284 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 04:51:09.971340 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:51:09.971389 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":68,"thread_start_us":91,"threads_started":1}
I20260928 04:51:09.971889 16168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b57cff9d0504fdcaf67730ddbbf7376" candidate_uuid: "8805768fa4594f3fb3e5620d1a5641a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48a41c7e18d14dc0b4c5994622fe1214" is_pre_election: true
I20260928 04:51:09.971961 15900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b57cff9d0504fdcaf67730ddbbf7376" candidate_uuid: "8805768fa4594f3fb3e5620d1a5641a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905806324bdb4c0ca488ff07d4ced7ea" is_pre_election: true
I20260928 04:51:09.972040 16168 raft_consensus.cc:2454] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8805768fa4594f3fb3e5620d1a5641a6 in term 0.
I20260928 04:51:09.972113 15900 raft_consensus.cc:2454] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8805768fa4594f3fb3e5620d1a5641a6 in term 0.
I20260928 04:51:09.972316 15968 leader_election.cc:302] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48a41c7e18d14dc0b4c5994622fe1214, 8805768fa4594f3fb3e5620d1a5641a6; no voters:
I20260928 04:51:09.972445 16236 raft_consensus.cc:2781] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:51:09.972498 16236 raft_consensus.cc:482] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:51:09.972527 16236 raft_consensus.cc:3037] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:51:09.973155 16236 raft_consensus.cc:504] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:09.973311 16236 leader_election.cc:288] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [CANDIDATE]: Term 1 election: Requested vote from peers 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545), 905806324bdb4c0ca488ff07d4ced7ea (127.0.98.1:34037)
I20260928 04:51:09.973470 15900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b57cff9d0504fdcaf67730ddbbf7376" candidate_uuid: "8805768fa4594f3fb3e5620d1a5641a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905806324bdb4c0ca488ff07d4ced7ea"
I20260928 04:51:09.973460 16168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b57cff9d0504fdcaf67730ddbbf7376" candidate_uuid: "8805768fa4594f3fb3e5620d1a5641a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48a41c7e18d14dc0b4c5994622fe1214"
I20260928 04:51:09.973537 16168 raft_consensus.cc:3037] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:51:09.973537 15900 raft_consensus.cc:3037] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:51:09.974377 15900 raft_consensus.cc:2454] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8805768fa4594f3fb3e5620d1a5641a6 in term 1.
I20260928 04:51:09.974377 16168 raft_consensus.cc:2454] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8805768fa4594f3fb3e5620d1a5641a6 in term 1.
I20260928 04:51:09.974543 15968 leader_election.cc:302] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48a41c7e18d14dc0b4c5994622fe1214, 8805768fa4594f3fb3e5620d1a5641a6; no voters:
I20260928 04:51:09.974648 16236 raft_consensus.cc:2781] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:51:09.974797 16236 raft_consensus.cc:686] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 LEADER]: Becoming Leader. State: Replica: 8805768fa4594f3fb3e5620d1a5641a6, State: Running, Role: LEADER
I20260928 04:51:09.974898 16236 consensus_queue.cc:237] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:09.975639 15754 catalog_manager.cc:5495] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8805768fa4594f3fb3e5620d1a5641a6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8805768fa4594f3fb3e5620d1a5641a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } health_report { overall_health: HEALTHY } } }
I20260928 04:51:09.982771 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.980807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.0.98.2:39971)
0928 04:51:09.980843 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:09.980844 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:09.980864 (+ 20us) server_negotiation.cc:407] Connection header received
0928 04:51:09.980913 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:09.980913 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:09.980926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:09.980983 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:51:09.981065 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.981534 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.982164 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:09.982350 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:09.982398 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:09.982523 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:51:09.982616 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:51:09.982674 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:09.982706 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 04:51:09.982722 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:51:09.984505 16168 raft_consensus.cc:1261] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214 [term 1 FOLLOWER]: Refusing update from remote peer 8805768fa4594f3fb3e5620d1a5641a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:51:09.984530 15900 raft_consensus.cc:1261] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Refusing update from remote peer 8805768fa4594f3fb3e5620d1a5641a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:51:09.984838 16236 consensus_queue.cc:1035] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:51:09.984963 16238 consensus_queue.cc:1035] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:51:09.987282 16241 mvcc.cc:205] Tried to move back new op lower bound from 7334179102653689856 to 7334179102617686016. Current Snapshot: MvccSnapshot[applied={T|T < 7334179102653689856}]
I20260928 04:51:09.987840 16243 mvcc.cc:205] Tried to move back new op lower bound from 7334179102653689856 to 7334179102617686016. Current Snapshot: MvccSnapshot[applied={T|T < 7334179102653689856}]
I20260928 04:51:09.989748 15754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:51:09.989840 15754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:51:09.990532 15754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Sending DeleteTablet for 3 replicas of tablet 6b57cff9d0504fdcaf67730ddbbf7376
I20260928 04:51:09.990903 15880 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b57cff9d0504fdcaf67730ddbbf7376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:51:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44754
I20260928 04:51:09.990928 16148 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b57cff9d0504fdcaf67730ddbbf7376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:51:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47914
I20260928 04:51:09.990927 16014 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b57cff9d0504fdcaf67730ddbbf7376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:51:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58150
I20260928 04:51:09.991123 16251 tablet_replica.cc:330] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: stopping tablet replica
I20260928 04:51:09.991255 16251 raft_consensus.cc:2229] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:51:09.991293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 15817
I20260928 04:51:09.991930 16252 tablet_replica.cc:330] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: stopping tablet replica
I20260928 04:51:09.992022 16252 raft_consensus.cc:2229] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:51:09.992113 16252 raft_consensus.cc:2258] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:51:09.992308 16253 tablet_replica.cc:330] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: stopping tablet replica
I20260928 04:51:09.992393 16253 raft_consensus.cc:2229] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:51:09.992504 16253 raft_consensus.cc:2258] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:51:09.992872 16252 ts_tablet_manager.cc:1905] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:51:09.994242 16252 ts_tablet_manager.cc:1918] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:51:09.994325 16252 log.cc:1196] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/6b57cff9d0504fdcaf67730ddbbf7376
I20260928 04:51:09.994563 16252 ts_tablet_manager.cc:1939] T 6b57cff9d0504fdcaf67730ddbbf7376 P 48a41c7e18d14dc0b4c5994622fe1214: Deleting consensus metadata
I20260928 04:51:09.996878 16253 ts_tablet_manager.cc:1905] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:51:09.997612 15740 catalog_manager.cc:4841] TS 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545): tablet 6b57cff9d0504fdcaf67730ddbbf7376 (table test-table [id=1dcfb66266a54f8aada42a34a63a875c]) successfully deleted
W20260928 04:51:09.997840 15739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34037 (error 108)
I20260928 04:51:09.998118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 15951
I20260928 04:51:09.998339 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:09.998153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.0.1:44758)
0928 04:51:09.998232 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:09.998269 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 04:51:09.998824 15739 catalog_manager.cc:4584] TS 905806324bdb4c0ca488ff07d4ced7ea (127.0.98.1:34037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b57cff9d0504fdcaf67730ddbbf7376: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34037: connect: Connection refused (error 111)
I20260928 04:51:10.002830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 16085
I20260928 04:51:10.002930 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:10.002815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39971 (local address 127.0.0.1:58176)
0928 04:51:10.002866 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:10.002893 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:51:10.007771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 15723
I20260928 04:51:30.013052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33349
--webserver_interface=127.0.98.62
--webserver_port=40877
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:51:30.091056 16255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:30.091288 16255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:30.091326 16255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:30.092418 16255 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 04:51:30.092496 16255 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 04:51:30.092518 16255 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 04:51:30.092535 16255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:51:30.092566 16255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:30.092589 16255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:30.092613 16255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:51:30.092643 16255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:51:30.094058 16255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:30.094312 16255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:30.094530 16255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:51:30.102957 16261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:30.103010 16264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:30.103025 16262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:30.103050 16255 server_base.cc:1034] running on GCE node
I20260928 04:51:30.103396 16255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:30.103680 16255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:30.104833 16255 hybrid_clock.cc:648] HybridClock initialized: now 1790571090104822 us; error 29 us; skew 500 ppm
Sep 28 04:51:30 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:30.107465 16255 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:51:30.107896 16255 webserver.cc:460] Webserver started at http://127.0.98.62:40877/ using document root <none> and password file <none>
I20260928 04:51:30.108084 16255 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:30.108134 16255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:30.109403 16255 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:51:30.110009 16271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:30.110179 16255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:51:30.110291 16255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "1cbc006c0baca004b0a0085d91203d7a"
server_key_iv: "ba1b861d7131e7ee197b35b2c20fd273"
server_key_version: "encryptionkey@0"
I20260928 04:51:30.110651 16255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:30.122860 16255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:30.124577 16255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:30.124729 16255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:30.128856 16255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33349
I20260928 04:51:30.128894 16323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33349 every 8 connection(s)
I20260928 04:51:30.129398 16255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:51:30.130748 16324 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:51:30.131402 16324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Bootstrap starting.
I20260928 04:51:30.137912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 16255
I20260928 04:51:30.138180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34037
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33349
--webserver_port=38959
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:51:30.139606 16324 log.cc:824] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:30.141542 16324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:51:30.141834 16324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Bootstrap complete.
I20260928 04:51:30.144047 16324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:30.144253 16324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ed38cc827b4cf2a8b17dbd9f4eb1af, State: Initialized, Role: FOLLOWER
I20260928 04:51:30.144404 16324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:30.144475 16324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:51:30.144526 16324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:51:30.144577 16324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:51:30.145570 16324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:30.145651 16324 leader_election.cc:302] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29ed38cc827b4cf2a8b17dbd9f4eb1af; no voters:
I20260928 04:51:30.145896 16324 leader_election.cc:288] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 04:51:30.145928 16328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:51:30.146036 16328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 2 LEADER]: Becoming Leader. State: Replica: 29ed38cc827b4cf2a8b17dbd9f4eb1af, State: Running, Role: LEADER
I20260928 04:51:30.146117 16328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:30.146152 16324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:51:30.146404 16328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ed38cc827b4cf2a8b17dbd9f4eb1af. Latest consensus state: current_term: 2 leader_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } } }
I20260928 04:51:30.146493 16328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: This master's current role is: LEADER
I20260928 04:51:30.146404 16329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } } }
I20260928 04:51:30.146539 16329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: This master's current role is: LEADER
I20260928 04:51:30.147181 16332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:51:30.148026 16332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:51:30.148161 16332 catalog_manager.cc:1259] Loaded cluster ID: f1bbe07742b24d99bc958dc1583ad158
I20260928 04:51:30.148353 16332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:51:30.149497 16332 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:51:30.149660 16332 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Loaded TSK: 0
I20260928 04:51:30.167747 16332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Generated new TSK 1
I20260928 04:51:30.167833 16332 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:51:30.218575 16326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:30.218804 16326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:30.218844 16326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:30.219977 16326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:30.220039 16326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:30.220082 16326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:51:30.221561 16326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38959
--tserver_master_addrs=127.0.98.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:30.221838 16326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:30.222054 16326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:51:30.227032 16351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:30.227047 16354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:30.227054 16352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:30.227382 16326 server_base.cc:1034] running on GCE node
I20260928 04:51:30.227600 16326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:30.227839 16326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:30.229001 16326 hybrid_clock.cc:648] HybridClock initialized: now 1790571090228978 us; error 40 us; skew 500 ppm
Sep 28 04:51:30 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:30.231420 16326 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:51:30.231868 16326 webserver.cc:460] Webserver started at http://127.0.98.1:38959/ using document root <none> and password file <none>
I20260928 04:51:30.232079 16326 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:30.232155 16326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:30.233500 16326 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:51:30.234174 16361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:30.234467 16326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:51:30.234579 16326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "905806324bdb4c0ca488ff07d4ced7ea"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "23919486946cd82e196e65f8cee1d02c"
server_key_iv: "8e6a3a9ea9a3382772afb19ee47d176a"
server_key_version: "encryptionkey@0"
I20260928 04:51:30.234921 16326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:30.262679 16326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:30.265014 16326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:30.265202 16326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:30.265435 16326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:51:30.265931 16368 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:51:30.267346 16326 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:51:30.267396 16326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:51:30.267431 16326 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:51:30.268077 16326 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:51:30.268133 16326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:51:30.268173 16368 tablet_bootstrap.cc:492] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Bootstrap starting.
I20260928 04:51:30.275553 16326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34037
I20260928 04:51:30.275689 16475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34037 every 8 connection(s)
I20260928 04:51:30.276054 16326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:51:30.279181 16368 log.cc:824] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:30.281551 16368 tablet_bootstrap.cc:492] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:51:30.281840 16368 tablet_bootstrap.cc:492] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Bootstrap complete.
I20260928 04:51:30.282812 16368 ts_tablet_manager.cc:1397] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260928 04:51:30.283144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 16326
I20260928 04:51:30.283429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39971
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33349
--webserver_port=46287
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 04:51:30 dist-test-slave-jccs krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:51:30.284427 16368 raft_consensus.cc:348] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:30.284806 16368 raft_consensus.cc:729] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905806324bdb4c0ca488ff07d4ced7ea, State: Initialized, Role: FOLLOWER
I20260928 04:51:30.284971 16368 consensus_queue.cc:260] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } }
I20260928 04:51:30.285254 16368 ts_tablet_manager.cc:1428] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260928 04:51:30.288169 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.276891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45453 (local address 127.0.98.62:33349)
0928 04:51:30.277318 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.277330 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.277380 (+ 50us) server_negotiation.cc:407] Connection header received
0928 04:51:30.277966 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.277974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.278035 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.278215 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:30.278901 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.280200 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.281296 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.281889 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.282050 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.284344 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:30.284347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:30.284348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:30.284358 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:30.286450 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:30.286879 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:30.286882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:30.286883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:30.286972 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:30.287228 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:30.287229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:30.287230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:30.287447 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:30.287549 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.287855 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.287996 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":98,"threads_started":1}
I20260928 04:51:30.288169 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.276758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:45453)
0928 04:51:30.277110 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:30.277148 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:30.277716 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:30.278255 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:30.278259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:30.278588 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:51:30.278765 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.278769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.280345 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.280346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.281142 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.281144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.282052 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.282053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.282154 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.282703 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:51:30.282711 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:51:30.284241 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:51:30.286535 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:30.286541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:30.286543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:30.286764 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:30.287053 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:30.287054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:30.287054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:30.287161 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:30.287528 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:51:30.287547 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:51:30.287675 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 04:51:30.287778 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 04:51:30.287977 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":90,"threads_started":1}
I20260928 04:51:30.289036 16476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33349
I20260928 04:51:30.289132 16476 heartbeater.cc:464] Registering TS with master...
I20260928 04:51:30.289316 16476 heartbeater.cc:511] Master 127.0.98.62:33349 requested a full tablet report, sending...
I20260928 04:51:30.289955 16288 ts_manager.cc:194] Registered new tserver with Master: 905806324bdb4c0ca488ff07d4ced7ea (127.0.98.1:34037)
I20260928 04:51:30.290915 16288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45453
I20260928 04:51:30.291664 16476 heartbeater.cc:503] Master 127.0.98.62:33349 was elected leader, sending a full tablet report...
I20260928 04:51:30.296526 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.290486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.0.1:53820)
0928 04:51:30.290623 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:30.290662 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:30.290728 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:30.293480 (+ 2752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:30.293482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:30.293485 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:51:30.293527 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.293528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.294752 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.294752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.295498 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.295499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.296299 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.296299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.296380 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.296389 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:51:30.296443 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:51:30.296477 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260928 04:51:30.296568 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.290554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.0.98.1:34037)
0928 04:51:30.293313 (+ 2759us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.293318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.293327 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:51:30.293360 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.293362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.293378 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.293463 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:51:30.293580 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.294674 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.295581 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.296223 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.296297 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.296371 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.296468 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.296503 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2724,"thread_start_us":69,"threads_started":1}
I20260928 04:51:30.296923 16410 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b57cff9d0504fdcaf67730ddbbf7376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:51:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53820
I20260928 04:51:30.297439 16485 tablet_replica.cc:330] stopping tablet replica
I20260928 04:51:30.297564 16485 raft_consensus.cc:2229] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:51:30.297646 16485 raft_consensus.cc:2258] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:51:30.297921 16485 ts_tablet_manager.cc:1905] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:51:30.299774 16485 ts_tablet_manager.cc:1918] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:51:30.299863 16485 log.cc:1196] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal/wals/6b57cff9d0504fdcaf67730ddbbf7376
I20260928 04:51:30.300071 16485 ts_tablet_manager.cc:1939] T 6b57cff9d0504fdcaf67730ddbbf7376 P 905806324bdb4c0ca488ff07d4ced7ea: Deleting consensus metadata
I20260928 04:51:30.300493 16272 catalog_manager.cc:4841] TS 905806324bdb4c0ca488ff07d4ced7ea (127.0.98.1:34037): tablet 6b57cff9d0504fdcaf67730ddbbf7376 (table test-table [id=1dcfb66266a54f8aada42a34a63a875c]) successfully deleted
W20260928 04:51:30.364569 16481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:30.364778 16481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:30.364815 16481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:30.365988 16481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:30.366048 16481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:30.366092 16481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:51:30.367542 16481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46287
--tserver_master_addrs=127.0.98.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:30.367790 16481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:30.368023 16481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:51:30.372804 16490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:30.372779 16491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:30.372788 16493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:30.373001 16481 server_base.cc:1034] running on GCE node
I20260928 04:51:30.373363 16481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:30.373585 16481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:30.374739 16481 hybrid_clock.cc:648] HybridClock initialized: now 1790571090374721 us; error 29 us; skew 500 ppm
Sep 28 04:51:30 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:30.376999 16481 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:51:30.377486 16481 webserver.cc:460] Webserver started at http://127.0.98.2:46287/ using document root <none> and password file <none>
I20260928 04:51:30.377700 16481 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:30.377784 16481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:30.379127 16481 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:51:30.379818 16500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:30.379998 16481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:51:30.380096 16481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "8805768fa4594f3fb3e5620d1a5641a6"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "77be6f12a0354ec8682831656da3f083"
server_key_iv: "134a9c3105daef761047429557c0df98"
server_key_version: "encryptionkey@0"
I20260928 04:51:30.380419 16481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:30.389812 16481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:30.391435 16481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:30.391574 16481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:30.391762 16481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:51:30.392133 16507 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 04:51:30.393404 16507 ts_tablet_manager.cc:1852] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 04:51:30.393648 16507 ts_tablet_manager.cc:1905] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:51:30.395818 16507 ts_tablet_manager.cc:1918] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:51:30.395916 16507 log.cc:1196] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/6b57cff9d0504fdcaf67730ddbbf7376
I20260928 04:51:30.396176 16507 ts_tablet_manager.cc:1939] T 6b57cff9d0504fdcaf67730ddbbf7376 P 8805768fa4594f3fb3e5620d1a5641a6: Deleting consensus metadata
I20260928 04:51:30.396378 16481 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 04:51:30.396440 16481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 04:51:30.396487 16481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:51:30.396512 16481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:30.402318 16481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39971
I20260928 04:51:30.402369 16614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39971 every 8 connection(s)
I20260928 04:51:30.402639 16481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:51:30 dist-test-slave-jccs krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:51:30.409027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 16481
I20260928 04:51:30.409369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36545
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33349
--webserver_port=45375
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:51:30.411279 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.403270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40149 (local address 127.0.98.62:33349)
0928 04:51:30.403343 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.403345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.403617 (+ 272us) server_negotiation.cc:407] Connection header received
0928 04:51:30.404077 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.404079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.404089 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.404144 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:30.404723 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.405272 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.406208 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.406406 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.406458 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.408437 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:30.408438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:30.408438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:30.408445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:30.409962 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:30.410353 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:30.410354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:30.410355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:30.410419 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:30.410659 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:30.410660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:30.410661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:30.410771 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:30.410838 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.411107 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.411192 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:51:30.411433 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.403268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:40149)
0928 04:51:30.403464 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:30.403515 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:30.403969 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:30.404217 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:30.404221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:30.404469 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:51:30.404641 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.404644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.405347 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.405348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.406136 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.406138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.406476 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.406477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.406571 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.407036 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:51:30.407044 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:51:30.408345 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:51:30.410044 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:30.410051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:30.410053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:30.410263 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:30.410494 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:30.410495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:30.410495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:30.410594 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:30.410842 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:51:30.410848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:51:30.410982 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 04:51:30.411076 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 04:51:30.411220 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":124,"threads_started":1}
I20260928 04:51:30.411891 16615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33349
I20260928 04:51:30.412000 16615 heartbeater.cc:464] Registering TS with master...
I20260928 04:51:30.412165 16615 heartbeater.cc:511] Master 127.0.98.62:33349 requested a full tablet report, sending...
I20260928 04:51:30.412557 16288 ts_manager.cc:194] Registered new tserver with Master: 8805768fa4594f3fb3e5620d1a5641a6 (127.0.98.2:39971)
I20260928 04:51:30.413153 16288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40149
W20260928 04:51:30.488277 16618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:30.488509 16618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:30.488548 16618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:30.489686 16618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:30.489722 16618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:30.489748 16618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:51:30.491039 16618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45375
--tserver_master_addrs=127.0.98.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:30.491199 16618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:30.491348 16618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:51:30.496032 16618 server_base.cc:1034] running on GCE node
W20260928 04:51:30.496011 16626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:30.496016 16624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:30.496003 16623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:30.496413 16618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:30.496667 16618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:30.497819 16618 hybrid_clock.cc:648] HybridClock initialized: now 1790571090497807 us; error 25 us; skew 500 ppm
Sep 28 04:51:30 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:30.500113 16618 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:51:30.500447 16618 webserver.cc:460] Webserver started at http://127.0.98.3:45375/ using document root <none> and password file <none>
I20260928 04:51:30.500607 16618 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:30.500653 16618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:30.501919 16618 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:51:30.502445 16633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:30.502625 16618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:30.502691 16618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "48a41c7e18d14dc0b4c5994622fe1214"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "91a4e36d381f49f6a595f9d1a19c82e1"
server_key_iv: "0ccc24ffd244742426b2a726ce3440cd"
server_key_version: "encryptionkey@0"
I20260928 04:51:30.502946 16618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:30.520787 16618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:30.523023 16618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:30.523125 16618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:30.523314 16618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:51:30.523581 16618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:51:30.523610 16618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:30.523631 16618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:51:30.523643 16618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:30.529528 16618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36545
I20260928 04:51:30.529585 16746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36545 every 8 connection(s)
I20260928 04:51:30.529883 16618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:51:30.534325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 16618
Sep 28 04:51:30 dist-test-slave-jccs krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:51:30.539494 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.530566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38917 (local address 127.0.98.62:33349)
0928 04:51:30.530655 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.530658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.530865 (+ 207us) server_negotiation.cc:407] Connection header received
0928 04:51:30.531302 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.531303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.531313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.531365 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:30.532203 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.532674 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.533672 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.533862 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.533922 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.536364 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:30.536365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:30.536366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:30.536373 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:30.538096 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:30.538472 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:30.538473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:30.538473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:30.538538 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:30.538795 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:30.538796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:30.538796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:30.538934 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:30.539006 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.539313 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.539394 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:51:30.539695 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.530555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:38917)
0928 04:51:30.530728 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:30.530760 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:30.531225 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:30.531414 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:30.531417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:30.531947 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:51:30.532128 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.532132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.532746 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.532747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.533605 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.533607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.533928 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.533929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.534020 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.534551 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:51:30.534560 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:51:30.536277 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:51:30.538174 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:30.538179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:30.538180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:30.538387 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:30.538610 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:30.538611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:30.538611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:30.538714 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:30.539000 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:51:30.539005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:51:30.539162 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 04:51:30.539268 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 04:51:30.539454 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260928 04:51:30.540144 16747 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33349
I20260928 04:51:30.540237 16747 heartbeater.cc:464] Registering TS with master...
I20260928 04:51:30.540426 16747 heartbeater.cc:511] Master 127.0.98.62:33349 requested a full tablet report, sending...
I20260928 04:51:30.540812 16288 ts_manager.cc:194] Registered new tserver with Master: 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545)
I20260928 04:51:30.541340 16288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38917
I20260928 04:51:30.541728 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.534725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.0.98.62:33349)
0928 04:51:30.534853 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.534855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.534962 (+ 107us) server_negotiation.cc:407] Connection header received
0928 04:51:30.534985 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.534986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.534997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.535071 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:30.535841 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.536387 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.537281 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.537546 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.537596 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.538945 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:30.538946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:30.538947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:30.538952 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:30.540622 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:30.541000 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:30.541001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:30.541001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:30.541049 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:30.541251 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:30.541251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:30.541252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:30.541351 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:30.541529 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.541568 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.541643 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260928 04:51:30.542341 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:51:30.544660 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.542646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.0.98.62:33349)
0928 04:51:30.542704 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.542706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.542751 (+ 45us) server_negotiation.cc:407] Connection header received
0928 04:51:30.542803 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.542804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.542828 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.542883 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:51:30.542974 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.543385 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.544106 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.544281 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.544344 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.544465 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:51:30.544499 (+ 34us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 04:51:30.544603 (+ 104us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48510: token expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 04:51:30.544706 16751 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48510: token expired
W20260928 04:51:30.544799 16218 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:33349)
I20260928 04:51:30.550457 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.545006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.0.98.62:33349)
0928 04:51:30.545045 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.545047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.545055 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:51:30.545107 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.545108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.545116 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.545161 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:30.545663 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.546115 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.546844 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.547037 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.547107 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.548280 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:30.548281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:30.548282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:30.548288 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:30.549340 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:30.549707 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:30.549708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:30.549708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:30.549757 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:30.549971 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:30.549972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:30.549972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:30.550094 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:30.550191 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.550287 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.550359 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:51:30.551522 16288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:51:30.552925 16288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:51:30.554795 16410 tablet_service.cc:1467] Processing CreateTablet for tablet d226edd9cf754c619795a0a99e6287e4 (DEFAULT_TABLE table=test-table [id=352ae7c6b8f24576ac99a9bb7ff5631d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:51:30.554998 16410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d226edd9cf754c619795a0a99e6287e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:51:30.556260 16368 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Bootstrap starting.
I20260928 04:51:30.556612 16368 tablet_bootstrap.cc:654] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Neither blocks nor log segments found. Creating new log.
I20260928 04:51:30.557206 16368 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: No bootstrap required, opened a new log
I20260928 04:51:30.557258 16368 ts_tablet_manager.cc:1397] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 04:51:30.557372 16368 raft_consensus.cc:348] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:30.557456 16368 raft_consensus.cc:374] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:51:30.557505 16368 raft_consensus.cc:729] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905806324bdb4c0ca488ff07d4ced7ea, State: Initialized, Role: FOLLOWER
I20260928 04:51:30.557556 16368 consensus_queue.cc:260] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:30.557660 16368 ts_tablet_manager.cc:1428] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:30.558845 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.554547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39971 (local address 127.0.0.1:49368)
0928 04:51:30.554589 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:30.554600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:30.554631 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:30.555196 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:30.555197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:30.555199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:51:30.555232 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.555234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.556795 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.556796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.557944 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.557946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.558633 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.558634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.558677 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.558679 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:51:30.558725 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:51:30.558784 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":2}
I20260928 04:51:30.558943 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.554756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49368 (local address 127.0.98.2:39971)
0928 04:51:30.555029 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.555033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.555042 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:51:30.555089 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.555091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.555105 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.555187 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:51:30.555314 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.556732 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.558021 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.558558 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.558622 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.558663 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.558791 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.558834 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
I20260928 04:51:30.559427 16549 tablet_service.cc:1467] Processing CreateTablet for tablet d226edd9cf754c619795a0a99e6287e4 (DEFAULT_TABLE table=test-table [id=352ae7c6b8f24576ac99a9bb7ff5631d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:51:30.559623 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.554565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.0.1:57840)
0928 04:51:30.554709 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:30.554717 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:30.554749 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:30.555455 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:30.555455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:30.555457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:51:30.555487 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.555488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.556698 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.556699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.557951 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.557952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.559384 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.559385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.559455 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.559459 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:51:30.559523 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 04:51:30.559557 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":74,"thread_start_us":78,"threads_started":1}
I20260928 04:51:30.559665 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.554745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.0.98.3:36545)
0928 04:51:30.555311 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.555314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.555323 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:51:30.555355 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.555357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.555370 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.555446 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:51:30.555553 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.556613 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.558690 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.559310 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.559364 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.559393 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.559538 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.559565 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":137,"threads_started":1}
I20260928 04:51:30.559739 16549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d226edd9cf754c619795a0a99e6287e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:51:30.560245 16681 tablet_service.cc:1467] Processing CreateTablet for tablet d226edd9cf754c619795a0a99e6287e4 (DEFAULT_TABLE table=test-table [id=352ae7c6b8f24576ac99a9bb7ff5631d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:51:30.560518 16681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d226edd9cf754c619795a0a99e6287e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:51:30.561481 16507 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Bootstrap starting.
I20260928 04:51:30.562170 16507 tablet_bootstrap.cc:654] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Neither blocks nor log segments found. Creating new log.
I20260928 04:51:30.562458 16507 log.cc:824] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:30.563287 16507 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: No bootstrap required, opened a new log
I20260928 04:51:30.563347 16507 ts_tablet_manager.cc:1397] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:51:30.563519 16758 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Bootstrap starting.
I20260928 04:51:30.564042 16758 tablet_bootstrap.cc:654] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Neither blocks nor log segments found. Creating new log.
I20260928 04:51:30.564510 16758 log.cc:824] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:30.564843 16507 raft_consensus.cc:348] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:30.564986 16507 raft_consensus.cc:374] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:51:30.565029 16507 raft_consensus.cc:729] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8805768fa4594f3fb3e5620d1a5641a6, State: Initialized, Role: FOLLOWER
I20260928 04:51:30.565143 16758 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: No bootstrap required, opened a new log
I20260928 04:51:30.565160 16507 consensus_queue.cc:260] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:30.565229 16758 ts_tablet_manager.cc:1397] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:51:30.565421 16507 ts_tablet_manager.cc:1428] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:51:30.565469 16615 heartbeater.cc:503] Master 127.0.98.62:33349 was elected leader, sending a full tablet report...
I20260928 04:51:30.566993 16758 raft_consensus.cc:348] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:30.567113 16758 raft_consensus.cc:374] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:51:30.567134 16758 raft_consensus.cc:729] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a41c7e18d14dc0b4c5994622fe1214, State: Initialized, Role: FOLLOWER
I20260928 04:51:30.567256 16758 consensus_queue.cc:260] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:30.567531 16758 ts_tablet_manager.cc:1428] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:51:30.567531 16747 heartbeater.cc:503] Master 127.0.98.62:33349 was elected leader, sending a full tablet report...
I20260928 04:51:30.646764 16761 raft_consensus.cc:482] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:51:30.646943 16761 raft_consensus.cc:504] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:30.647328 16761 leader_election.cc:288] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8805768fa4594f3fb3e5620d1a5641a6 (127.0.98.2:39971), 905806324bdb4c0ca488ff07d4ced7ea (127.0.98.1:34037)
I20260928 04:51:30.650230 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.647559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49617 (local address 127.0.98.1:34037)
0928 04:51:30.647625 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.647627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.647654 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:51:30.647706 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.647707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.647719 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.647776 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:51:30.647943 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.648480 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.649498 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.649940 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.649991 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.650000 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.650154 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.650181 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:51:30.650255 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.647524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.98.3:49617)
0928 04:51:30.647583 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:30.647594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:30.647636 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:30.647791 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:30.647792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:30.647794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:51:30.647870 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.647872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.648551 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.648552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.649416 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.649418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.650005 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.650006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.650082 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.650086 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:51:30.650144 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 04:51:30.650179 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 04:51:30.650576 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.647734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56265 (local address 127.0.98.2:39971)
0928 04:51:30.647806 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.647808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.647816 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:51:30.647840 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.647841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.647854 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.647923 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:51:30.648102 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.648734 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.649769 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.650227 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.650287 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.650297 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.650480 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.650522 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 04:51:30.650532 16430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d226edd9cf754c619795a0a99e6287e4" candidate_uuid: "48a41c7e18d14dc0b4c5994622fe1214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905806324bdb4c0ca488ff07d4ced7ea" is_pre_election: true
I20260928 04:51:30.650691 16430 raft_consensus.cc:2454] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48a41c7e18d14dc0b4c5994622fe1214 in term 0.
I20260928 04:51:30.650777 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.647524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39971 (local address 127.0.98.3:56265)
0928 04:51:30.647711 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:30.647722 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:30.647766 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:30.647943 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:30.647943 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:30.647945 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:51:30.648034 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.648035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.648828 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.648829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.649688 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:30.649690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.650300 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.650301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:30.650376 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.650381 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:51:30.650665 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 04:51:30.650709 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":22,"thread_start_us":90,"threads_started":1}
I20260928 04:51:30.650940 16634 leader_election.cc:302] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48a41c7e18d14dc0b4c5994622fe1214, 905806324bdb4c0ca488ff07d4ced7ea; no voters:
I20260928 04:51:30.651077 16761 raft_consensus.cc:2781] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:51:30.651137 16761 raft_consensus.cc:482] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:51:30.651105 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d226edd9cf754c619795a0a99e6287e4" candidate_uuid: "48a41c7e18d14dc0b4c5994622fe1214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8805768fa4594f3fb3e5620d1a5641a6" is_pre_election: true
I20260928 04:51:30.651165 16761 raft_consensus.cc:3037] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:51:30.651240 16569 raft_consensus.cc:2454] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48a41c7e18d14dc0b4c5994622fe1214 in term 0.
I20260928 04:51:30.651912 16761 raft_consensus.cc:504] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:30.652041 16761 leader_election.cc:288] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [CANDIDATE]: Term 1 election: Requested vote from peers 8805768fa4594f3fb3e5620d1a5641a6 (127.0.98.2:39971), 905806324bdb4c0ca488ff07d4ced7ea (127.0.98.1:34037)
I20260928 04:51:30.652170 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d226edd9cf754c619795a0a99e6287e4" candidate_uuid: "48a41c7e18d14dc0b4c5994622fe1214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8805768fa4594f3fb3e5620d1a5641a6"
I20260928 04:51:30.652179 16430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d226edd9cf754c619795a0a99e6287e4" candidate_uuid: "48a41c7e18d14dc0b4c5994622fe1214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905806324bdb4c0ca488ff07d4ced7ea"
I20260928 04:51:30.652244 16569 raft_consensus.cc:3037] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:51:30.652244 16430 raft_consensus.cc:3037] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:51:30.653136 16569 raft_consensus.cc:2454] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48a41c7e18d14dc0b4c5994622fe1214 in term 1.
W20260928 04:51:30.653132 16616 tablet.cc:2369] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:51:30.653482 16430 raft_consensus.cc:2454] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48a41c7e18d14dc0b4c5994622fe1214 in term 1.
I20260928 04:51:30.653506 16636 leader_election.cc:302] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48a41c7e18d14dc0b4c5994622fe1214, 8805768fa4594f3fb3e5620d1a5641a6; no voters:
I20260928 04:51:30.653708 16761 raft_consensus.cc:2781] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:51:30.653867 16761 raft_consensus.cc:686] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 1 LEADER]: Becoming Leader. State: Replica: 48a41c7e18d14dc0b4c5994622fe1214, State: Running, Role: LEADER
I20260928 04:51:30.653961 16761 consensus_queue.cc:237] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:30.654780 16287 catalog_manager.cc:5495] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "48a41c7e18d14dc0b4c5994622fe1214" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } health_report { overall_health: UNKNOWN } } }
I20260928 04:51:30.673269 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.671119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.0.98.3:36545)
0928 04:51:30.671172 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:30.671173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:30.671181 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:51:30.671212 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:30.671212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:30.671231 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:30.671307 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:51:30.671446 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.671859 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.672560 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:30.672738 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:30.672809 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:30.672960 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:51:30.673044 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:51:30.673131 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:30.673149 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 04:51:30.673169 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:51:30.674700 16430 raft_consensus.cc:1261] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Refusing update from remote peer 48a41c7e18d14dc0b4c5994622fe1214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:51:30.674705 16569 raft_consensus.cc:1261] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Refusing update from remote peer 48a41c7e18d14dc0b4c5994622fe1214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:51:30.674997 16763 consensus_queue.cc:1035] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:51:30.675112 16761 consensus_queue.cc:1035] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:51:30.677068 16766 mvcc.cc:205] Tried to move back new op lower bound from 7334179187401412608 to 7334179187319230464. Current Snapshot: MvccSnapshot[applied={T|T < 7334179187401412608}]
I20260928 04:51:30.677886 16768 mvcc.cc:205] Tried to move back new op lower bound from 7334179187401412608 to 7334179187319230464. Current Snapshot: MvccSnapshot[applied={T|T < 7334179187401412608 or (T in {7334179187401412608})}]
I20260928 04:51:30.679294 16287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:51:30.679383 16287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:51:30.680012 16287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Sending DeleteTablet for 3 replicas of tablet d226edd9cf754c619795a0a99e6287e4
I20260928 04:51:30.680317 16410 tablet_service.cc:1514] Processing DeleteTablet for tablet d226edd9cf754c619795a0a99e6287e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:51:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53820
I20260928 04:51:30.680356 16549 tablet_service.cc:1514] Processing DeleteTablet for tablet d226edd9cf754c619795a0a99e6287e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:51:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49368
I20260928 04:51:30.680549 16485 tablet_replica.cc:330] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: stopping tablet replica
I20260928 04:51:30.680573 16775 tablet_replica.cc:330] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: stopping tablet replica
I20260928 04:51:30.680615 16485 raft_consensus.cc:2229] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:51:30.680672 16485 raft_consensus.cc:2258] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:51:30.680682 16775 raft_consensus.cc:2229] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:51:30.680770 16775 raft_consensus.cc:2258] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:51:30.680958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 16326
I20260928 04:51:30.680955 16681 tablet_service.cc:1514] Processing DeleteTablet for tablet d226edd9cf754c619795a0a99e6287e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:51:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57840
I20260928 04:51:30.681313 16485 ts_tablet_manager.cc:1905] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:51:30.681564 16776 tablet_replica.cc:330] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: stopping tablet replica
I20260928 04:51:30.681649 16776 raft_consensus.cc:2229] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:51:30.681744 16776 raft_consensus.cc:2258] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:51:30.682130 16776 ts_tablet_manager.cc:1905] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:51:30.684006 16776 ts_tablet_manager.cc:1918] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:51:30.684078 16776 log.cc:1196] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/d226edd9cf754c619795a0a99e6287e4
I20260928 04:51:30.684373 16776 ts_tablet_manager.cc:1939] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Deleting consensus metadata
I20260928 04:51:30.684736 16273 catalog_manager.cc:4841] TS 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545): tablet d226edd9cf754c619795a0a99e6287e4 (table test-table [id=352ae7c6b8f24576ac99a9bb7ff5631d]) successfully deleted
W20260928 04:51:30.686650 16272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34037 (error 108)
I20260928 04:51:30.686921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 16481
I20260928 04:51:30.687084 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.686966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.0.1:53830)
0928 04:51:30.687009 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:30.687048 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 04:51:30.689623 16272 catalog_manager.cc:4584] TS 905806324bdb4c0ca488ff07d4ced7ea (127.0.98.1:34037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d226edd9cf754c619795a0a99e6287e4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34037: connect: Connection refused (error 111)
I20260928 04:51:30.691325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 16618
I20260928 04:51:30.691373 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:30.691246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39971 (local address 127.0.0.1:49380)
0928 04:51:30.691307 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:30.691337 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 04:51:30.695849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 16255
I20260928 04:51:50.700829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33349
--webserver_interface=127.0.98.62
--webserver_port=40877
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:51:50.781854 16778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:50.782064 16778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:50.782087 16778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:50.783381 16778 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 04:51:50.783427 16778 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 04:51:50.783439 16778 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 04:51:50.783452 16778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:51:50.783494 16778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:50.783522 16778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:50.783538 16778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:51:50.783560 16778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:51:50.785032 16778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:50.785317 16778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:50.785549 16778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:51:50.790453 16786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:50.790453 16783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:50.790443 16784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:50.790679 16778 server_base.cc:1034] running on GCE node
I20260928 04:51:50.790922 16778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:50.791215 16778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:50.792356 16778 hybrid_clock.cc:648] HybridClock initialized: now 1790571110792345 us; error 28 us; skew 500 ppm
Sep 28 04:51:50 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:50.794785 16778 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:51:50.795221 16778 webserver.cc:460] Webserver started at http://127.0.98.62:40877/ using document root <none> and password file <none>
I20260928 04:51:50.795514 16778 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:50.795581 16778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:50.796921 16778 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:51:50.797636 16793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:50.797806 16778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:50.797900 16778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "1cbc006c0baca004b0a0085d91203d7a"
server_key_iv: "ba1b861d7131e7ee197b35b2c20fd273"
server_key_version: "encryptionkey@0"
I20260928 04:51:50.798249 16778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:50.822770 16778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:50.824890 16778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:50.825094 16778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:50.829104 16778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33349
I20260928 04:51:50.829209 16845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33349 every 8 connection(s)
I20260928 04:51:50.829591 16778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:51:50.831085 16846 sys_catalog.cc:263] Verifying existing consensus state
I20260928 04:51:50.831765 16846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Bootstrap starting.
I20260928 04:51:50.835992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 16778
I20260928 04:51:50.836287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34037
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33349
--webserver_port=38959
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:51:50.840746 16846 log.cc:824] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:50.843127 16846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:51:50.843459 16846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Bootstrap complete.
I20260928 04:51:50.845779 16846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:50.845997 16846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ed38cc827b4cf2a8b17dbd9f4eb1af, State: Initialized, Role: FOLLOWER
I20260928 04:51:50.846199 16846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:50.846264 16846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:51:50.846314 16846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:51:50.846376 16846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 2 FOLLOWER]: Advancing to term 3
I20260928 04:51:50.847355 16846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:50.847479 16846 leader_election.cc:302] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29ed38cc827b4cf2a8b17dbd9f4eb1af; no voters:
I20260928 04:51:50.847684 16846 leader_election.cc:288] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 04:51:50.847726 16850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 3 FOLLOWER]: Leader election won for term 3
I20260928 04:51:50.847851 16850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [term 3 LEADER]: Becoming Leader. State: Replica: 29ed38cc827b4cf2a8b17dbd9f4eb1af, State: Running, Role: LEADER
I20260928 04:51:50.847960 16846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:51:50.847937 16850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } }
I20260928 04:51:50.848904 16852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ed38cc827b4cf2a8b17dbd9f4eb1af. Latest consensus state: current_term: 3 leader_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } } }
I20260928 04:51:50.848992 16852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: This master's current role is: LEADER
I20260928 04:51:50.849211 16860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:51:50.849380 16851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ed38cc827b4cf2a8b17dbd9f4eb1af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33349 } } }
I20260928 04:51:50.849445 16851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af [sys.catalog]: This master's current role is: LEADER
I20260928 04:51:50.850175 16860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:51:50.850369 16860 catalog_manager.cc:1259] Loaded cluster ID: f1bbe07742b24d99bc958dc1583ad158
I20260928 04:51:50.850452 16860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:51:50.851805 16860 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:51:50.851990 16860 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Loaded TSK: 0
I20260928 04:51:50.852042 16860 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29ed38cc827b4cf2a8b17dbd9f4eb1af: Loaded TSK: 1
F20260928 04:51:50.852285 16860 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7a6e4939ad google::LogMessage::Fail() at ??:0
@ 0x7f7a6e49688a google::LogMessage::SendToLog() at ??:0
@ 0x7f7a6e4934de google::LogMessage::Flush() at ??:0
@ 0x7f7a6e494a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7a70c00104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7a70bfdd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7a763f21cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7a763f248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7a763f2594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f7a763b8c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f7a763c2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7a703610d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7a70356954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7a711ba6db start_thread at ??:0
@ 0x7f7a6cf9161f clone at ??:0
W20260928 04:51:50.938081 16848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:50.938298 16848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:50.938325 16848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:50.939536 16848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:50.939584 16848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:50.939612 16848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:51:50.941151 16848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38959
--tserver_master_addrs=127.0.98.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:50.941450 16848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:50.941704 16848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:51:50.946776 16887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:50.946774 16888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:50.946981 16848 server_base.cc:1034] running on GCE node
W20260928 04:51:50.947063 16890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:50.947268 16848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:50.947554 16848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:50.948716 16848 hybrid_clock.cc:648] HybridClock initialized: now 1790571110948700 us; error 37 us; skew 500 ppm
Sep 28 04:51:50 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:50.951262 16848 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:51:50.951790 16848 webserver.cc:460] Webserver started at http://127.0.98.1:38959/ using document root <none> and password file <none>
I20260928 04:51:50.952018 16848 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:50.952061 16848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:50.953418 16848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:51:50.954169 16897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:50.954334 16848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:51:50.954435 16848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "905806324bdb4c0ca488ff07d4ced7ea"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "23919486946cd82e196e65f8cee1d02c"
server_key_iv: "8e6a3a9ea9a3382772afb19ee47d176a"
server_key_version: "encryptionkey@0"
I20260928 04:51:50.954797 16848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:50.966864 16848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:50.968933 16848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:50.969074 16848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:50.969369 16848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:51:50.969834 16904 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:51:50.971264 16848 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:51:50.971333 16848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:51:50.971359 16848 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:51:50.972009 16848 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:51:50.972050 16848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:51:50.972105 16904 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Bootstrap starting.
I20260928 04:51:50.974020 16904 log.cc:824] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:50.977257 16904 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:51:50.977558 16904 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Bootstrap complete.
I20260928 04:51:50.977722 16904 ts_tablet_manager.cc:1397] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260928 04:51:50.979118 16848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34037
I20260928 04:51:50.979164 17012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34037 every 8 connection(s)
I20260928 04:51:50.979460 16848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:51:50.979626 16904 raft_consensus.cc:348] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:50.980371 16904 raft_consensus.cc:729] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905806324bdb4c0ca488ff07d4ced7ea, State: Initialized, Role: FOLLOWER
I20260928 04:51:50.980511 16904 consensus_queue.cc:260] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:50.980782 16904 ts_tablet_manager.cc:1428] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260928 04:51:50.981516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 16848
I20260928 04:51:50.981863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39971
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33349
--webserver_port=46287
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:51:51.071374 17015 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:51:50.980174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:47961)
0928 04:51:50.980342 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:50.980393 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:50.980564 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:51.071154 (+ 90590us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
W20260928 04:51:51.071681 17013 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33349 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33349: Client connection negotiation failed: client connection to 127.0.98.62:33349: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 04:51:51.072028 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:51.071908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:57455)
0928 04:51:51.071962 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:51.072003 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 04:51:51.072403 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:51.072317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:39247)
0928 04:51:51.072365 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:51.072388 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 04:51:51.072558 17013 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 04:51:51.089881 17017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:51.090108 17017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:51.090148 17017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:51.091307 17017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:51.091372 17017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:51.091416 17017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:51:51.092875 17017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46287
--tserver_master_addrs=127.0.98.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:51.093143 17017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:51.093421 17017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:51:51.098318 17022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:51.098287 17023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:51.098287 17025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:51.098511 17017 server_base.cc:1034] running on GCE node
I20260928 04:51:51.098739 17017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:51.098994 17017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:51.100157 17017 hybrid_clock.cc:648] HybridClock initialized: now 1790571111100144 us; error 25 us; skew 500 ppm
Sep 28 04:51:51 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:51.102617 17017 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:51:51.102988 17017 webserver.cc:460] Webserver started at http://127.0.98.2:46287/ using document root <none> and password file <none>
I20260928 04:51:51.103168 17017 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:51.103209 17017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:51.104416 17017 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:51:51.105026 17032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:51.105258 17017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:51:51.105360 17017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "8805768fa4594f3fb3e5620d1a5641a6"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "77be6f12a0354ec8682831656da3f083"
server_key_iv: "134a9c3105daef761047429557c0df98"
server_key_version: "encryptionkey@0"
I20260928 04:51:51.105706 17017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:51.138414 17017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:51.140977 17017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:51.141127 17017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:51.141419 17017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:51:51.141866 17039 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 04:51:51.143335 17017 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 04:51:51.143394 17017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 04:51:51.143455 17017 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 04:51:51.144099 17017 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 04:51:51.144157 17017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 04:51:51.144228 17039 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Bootstrap starting.
I20260928 04:51:51.150974 17017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39971
I20260928 04:51:51.151057 17146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39971 every 8 connection(s)
I20260928 04:51:51.151383 17017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:51:51.152515 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:51.152088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:46497)
0928 04:51:51.152286 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:51.152360 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
W20260928 04:51:51.152748 17147 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33349 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33349: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
I20260928 04:51:51.153045 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:51.152948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:42503)
0928 04:51:51.152998 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:51.153023 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 04:51:51.153452 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:51.153352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:55315)
0928 04:51:51.153402 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:51.153424 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 04:51:51.153559 17147 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:51:51.155109 17039 log.cc:824] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:51.157258 17039 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:51:51.157577 17039 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Bootstrap complete.
I20260928 04:51:51.157787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 17017
I20260928 04:51:51.158052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36545
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33349
--webserver_port=45375
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:51:51.158844 17039 ts_tablet_manager.cc:1397] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260928 04:51:51.160308 17039 raft_consensus.cc:348] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:51.160557 17039 raft_consensus.cc:729] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8805768fa4594f3fb3e5620d1a5641a6, State: Initialized, Role: FOLLOWER
I20260928 04:51:51.160688 17039 consensus_queue.cc:260] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:51.160966 17039 ts_tablet_manager.cc:1428] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260928 04:51:51.161211 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:51.161097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:34727)
0928 04:51:51.161143 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:51.161168 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 04:51:51.161515 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:51.161429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:33765)
0928 04:51:51.161480 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:51.161501 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 04:51:51.240177 17151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:51:51.240411 17151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:51:51.240437 17151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:51:51.241580 17151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:51:51.241614 17151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:51:51.241639 17151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:51:51.242982 17151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45375
--tserver_master_addrs=127.0.98.62:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:51:51.243173 17151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:51:51.243348 17151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:51:51.248512 17151 server_base.cc:1034] running on GCE node
W20260928 04:51:51.248456 17160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:51.248456 17158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:51:51.248478 17157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:51:51.248898 17151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:51:51.249150 17151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:51:51.250350 17151 hybrid_clock.cc:648] HybridClock initialized: now 1790571111250332 us; error 33 us; skew 500 ppm
Sep 28 04:51:51 dist-test-slave-jccs krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:51:51.252807 17151 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:51:51.253271 17151 webserver.cc:460] Webserver started at http://127.0.98.3:45375/ using document root <none> and password file <none>
I20260928 04:51:51.253453 17151 fs_manager.cc:359] Metadata directory not provided
I20260928 04:51:51.253496 17151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:51:51.254827 17151 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:51:51.255441 17167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:51.255622 17151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:51:51.255729 17151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "48a41c7e18d14dc0b4c5994622fe1214"
format_stamp: "Formatted at 2026-09-28 04:51:09 on dist-test-slave-jccs"
server_key: "91a4e36d381f49f6a595f9d1a19c82e1"
server_key_iv: "0ccc24ffd244742426b2a726ce3440cd"
server_key_version: "encryptionkey@0"
I20260928 04:51:51.256112 17151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:51:51.275403 17151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:51:51.277487 17151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:51:51.277659 17151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:51:51.277908 17151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:51:51.278239 17151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:51:51.278293 17151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:51.278334 17151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:51:51.278363 17151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:51:51.284772 17151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36545
I20260928 04:51:51.284819 17280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36545 every 8 connection(s)
I20260928 04:51:51.285284 17151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:51:51.286350 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:51.285990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:55889)
0928 04:51:51.286145 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:51.286213 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
W20260928 04:51:51.286757 17281 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33349 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33349: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
I20260928 04:51:51.287034 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:51.286950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:47783)
0928 04:51:51.286993 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:51.287018 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 04:51:51.287268 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:51.287223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:49665)
0928 04:51:51.287242 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:51.287258 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260928 04:51:51.287385 17281 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 04:51:51.294164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 17151
I20260928 04:51:52.073220 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.072946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:41775)
0928 04:51:52.073075 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:52.073116 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":102,"threads_started":1}
I20260928 04:51:52.141491 17286 raft_consensus.cc:482] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:51:52.141610 17286 raft_consensus.cc:504] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:52.142001 17286 leader_election.cc:288] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8805768fa4594f3fb3e5620d1a5641a6 (127.0.98.2:39971), 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545)
Sep 28 04:51:52 dist-test-slave-jccs krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 28 04:51:52 dist-test-slave-jccs krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 04:51:52.160792 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.142262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57903 (local address 127.0.98.2:39971)
0928 04:51:52.142375 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:52.142388 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:52.142412 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:51:52.142562 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:52.142564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:52.142633 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:52.142768 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:52.143548 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:52.144880 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:52.146265 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:52.146866 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:52.146979 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:52.157025 (+ 10046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:52.157030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:52.157030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:52.157044 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:52.159016 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:52.159504 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:52.159507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:52.159508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:52.159631 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:52.159906 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:52.159907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:52.159908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:52.160168 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:52.160266 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:52.160551 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 04:51:52.160682 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":59,"threads_started":1}
I20260928 04:51:52.160804 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.142166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.98.1:60107)
0928 04:51:52.142288 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:52.142299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:52.142353 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:52.143184 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:52.143185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:52.143729 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:51:52.143778 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:52.143780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:52.145247 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:52.145248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:52.146156 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:52.146158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:52.146967 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:52.146969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:52.147137 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:52.154554 (+ 7417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:51:52.154567 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:51:52.156469 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:51:52.159136 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:52.159137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:52.159137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:52.159412 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:52.159741 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:52.159742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:52.159742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:52.159830 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:52.160245 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:51:52.160252 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:51:52.160389 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 04:51:52.160506 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 04:51:52.160693 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":8,"thread_start_us":67,"threads_started":1}
I20260928 04:51:52.160818 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.142305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60107 (local address 127.0.98.3:36545)
0928 04:51:52.142447 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:52.142454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:52.142472 (+ 18us) server_negotiation.cc:407] Connection header received
0928 04:51:52.142987 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:52.142989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:52.143064 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:52.143168 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:52.143878 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:52.145138 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:52.146239 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:52.146867 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:52.146979 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:52.156620 (+ 9641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:52.156626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:52.156627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:52.156643 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:52.159014 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:52.159512 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:52.159519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:52.159520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:52.159667 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:52.159907 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:52.159908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:52.159908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:52.160168 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:52.160265 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:52.160563 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 04:51:52.160714 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260928 04:51:52.161070 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.142147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39971 (local address 127.0.98.1:57903)
0928 04:51:52.142199 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:52.142210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:52.142264 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:52.142809 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:52.142811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:52.143096 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:51:52.143414 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:52.143424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:52.144974 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:52.144975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:52.146156 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:52.146158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:52.146969 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:52.146970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:52.147137 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:52.154554 (+ 7417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:51:52.154567 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:51:52.156895 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:51:52.159097 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:52.159103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:52.159105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:52.159412 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:52.159700 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:52.159701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:52.159702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:52.159830 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:52.160245 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:51:52.160252 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:51:52.160391 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 04:51:52.160499 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 04:51:52.160773 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":182}
I20260928 04:51:52.161478 17101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d226edd9cf754c619795a0a99e6287e4" candidate_uuid: "905806324bdb4c0ca488ff07d4ced7ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8805768fa4594f3fb3e5620d1a5641a6" is_pre_election: true
I20260928 04:51:52.161634 17101 raft_consensus.cc:2454] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905806324bdb4c0ca488ff07d4ced7ea in term 1.
W20260928 04:51:52.161850 16899 leader_election.cc:341] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545): Not found: Tablet not found: d226edd9cf754c619795a0a99e6287e4
I20260928 04:51:52.162048 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.161843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:48467)
0928 04:51:52.161980 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:52.162015 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260928 04:51:52.162047 16900 leader_election.cc:302] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8805768fa4594f3fb3e5620d1a5641a6, 905806324bdb4c0ca488ff07d4ced7ea; no voters: 48a41c7e18d14dc0b4c5994622fe1214
I20260928 04:51:52.162189 17286 raft_consensus.cc:2781] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 04:51:52.162276 17286 raft_consensus.cc:482] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:51:52.162312 17286 raft_consensus.cc:3037] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 1 FOLLOWER]: Advancing to term 2
I20260928 04:51:52.163430 17286 raft_consensus.cc:504] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:52.163586 17286 leader_election.cc:288] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [CANDIDATE]: Term 2 election: Requested vote from peers 8805768fa4594f3fb3e5620d1a5641a6 (127.0.98.2:39971), 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545)
I20260928 04:51:52.163776 17101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d226edd9cf754c619795a0a99e6287e4" candidate_uuid: "905806324bdb4c0ca488ff07d4ced7ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8805768fa4594f3fb3e5620d1a5641a6"
I20260928 04:51:52.163861 17101 raft_consensus.cc:3037] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 1 FOLLOWER]: Advancing to term 2
W20260928 04:51:52.163978 16899 leader_election.cc:341] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545): Not found: Tablet not found: d226edd9cf754c619795a0a99e6287e4
I20260928 04:51:52.164932 17101 raft_consensus.cc:2454] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905806324bdb4c0ca488ff07d4ced7ea in term 2.
I20260928 04:51:52.165098 16900 leader_election.cc:302] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8805768fa4594f3fb3e5620d1a5641a6, 905806324bdb4c0ca488ff07d4ced7ea; no voters: 48a41c7e18d14dc0b4c5994622fe1214
I20260928 04:51:52.165207 17286 raft_consensus.cc:2781] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 2 FOLLOWER]: Leader election won for term 2
I20260928 04:51:52.165329 17286 raft_consensus.cc:686] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [term 2 LEADER]: Becoming Leader. State: Replica: 905806324bdb4c0ca488ff07d4ced7ea, State: Running, Role: LEADER
I20260928 04:51:52.165441 17286 consensus_queue.cc:237] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:52.165652 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.165513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:48831)
0928 04:51:52.165567 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:52.165595 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 04:51:52.287972 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.287728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:45207)
0928 04:51:52.287877 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:52.287927 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
W20260928 04:51:52.652539 16899 consensus_peers.cc:487] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea -> Peer 48a41c7e18d14dc0b4c5994622fe1214 (127.0.98.3:36545): Couldn't send request to peer 48a41c7e18d14dc0b4c5994622fe1214. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d226edd9cf754c619795a0a99e6287e4. This is attempt 1: this message will repeat every 5th retry.
I20260928 04:51:52.724716 17101 raft_consensus.cc:1261] T d226edd9cf754c619795a0a99e6287e4 P 8805768fa4594f3fb3e5620d1a5641a6 [term 2 FOLLOWER]: Refusing update from remote peer 905806324bdb4c0ca488ff07d4ced7ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 04:51:52.725131 17286 consensus_queue.cc:1035] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 04:51:52.726046 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.725898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:33389)
0928 04:51:52.725978 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:52.726012 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":42,"threads_started":1}
I20260928 04:51:52.726931 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.726831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:57253)
0928 04:51:52.726877 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:52.726904 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 04:51:52.726986 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:52.726853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:50477)
0928 04:51:52.726928 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:52.726955 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57,"thread_start_us":41,"threads_started":1}
I20260928 04:51:53.101855 17301 ts_tablet_manager.cc:927] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Initiating tablet copy from peer 905806324bdb4c0ca488ff07d4ced7ea (127.0.98.1:34037)
I20260928 04:51:53.102070 17301 tablet_copy_client.cc:316] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:34037
Sep 28 04:51:53 dist-test-slave-jccs krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 04:51:53.109835 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:53.102350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.98.3:45019)
0928 04:51:53.102519 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:53.102575 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 04:51:53.102643 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:51:53.102784 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:51:53.102788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:51:53.103077 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:51:53.103134 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:53.103135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:53.104018 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:53.104018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:53.104592 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:51:53.104594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:51:53.104997 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:51:53.104998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:51:53.105050 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:53.105597 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:51:53.105601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:51:53.106843 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:51:53.108598 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:53.108604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:53.108606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:53.108797 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:53.109043 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:51:53.109044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:51:53.109044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:51:53.109099 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:51:53.109470 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:51:53.109477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:51:53.109537 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 04:51:53.109657 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 04:51:53.109758 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
I20260928 04:51:53.109834 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:53.102441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45019 (local address 127.0.98.1:34037)
0928 04:51:53.102591 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 04:51:53.102596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:51:53.102626 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:51:53.102685 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:51:53.102687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:51:53.102699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:51:53.102768 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:51:53.103200 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:53.103940 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:53.104659 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:51:53.104934 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:51:53.104985 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:51:53.106918 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:51:53.106924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:51:53.106925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:51:53.106930 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:51:53.108461 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:53.108904 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:53.108909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:53.108909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:53.108968 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:51:53.109168 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:51:53.109192 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:51:53.109192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:51:53.109391 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:51:53.109491 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 04:51:53.109694 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 04:51:53.109760 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260928 04:51:53.110275 16987 tablet_copy_service.cc:140] P 905806324bdb4c0ca488ff07d4ced7ea: Received BeginTabletCopySession request for tablet d226edd9cf754c619795a0a99e6287e4 from peer 48a41c7e18d14dc0b4c5994622fe1214 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45019)
I20260928 04:51:53.110366 16987 tablet_copy_service.cc:161] P 905806324bdb4c0ca488ff07d4ced7ea: Beginning new tablet copy session on tablet d226edd9cf754c619795a0a99e6287e4 from peer 48a41c7e18d14dc0b4c5994622fe1214 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45019: session id = 48a41c7e18d14dc0b4c5994622fe1214-d226edd9cf754c619795a0a99e6287e4
I20260928 04:51:53.110898 16987 tablet_copy_source_session.cc:205] T d226edd9cf754c619795a0a99e6287e4 P 905806324bdb4c0ca488ff07d4ced7ea: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 04:51:53.111799 17301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d226edd9cf754c619795a0a99e6287e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:51:53.114265 17301 tablet_copy_client.cc:791] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: tablet copy: Starting download of 0 data blocks...
I20260928 04:51:53.114429 17301 tablet_copy_client.cc:655] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: tablet copy: Starting download of 1 WAL segments...
I20260928 04:51:53.115191 17301 tablet_copy_client.cc:531] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 04:51:53.115810 17301 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Bootstrap starting.
I20260928 04:51:53.117056 17301 log.cc:824] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Log is configured to *not* fsync() on all Append() calls
I20260928 04:51:53.119328 17301 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 04:51:53.119560 17301 tablet_bootstrap.cc:492] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Bootstrap complete.
I20260928 04:51:53.119702 17301 ts_tablet_manager.cc:1397] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 04:51:53.120733 17301 raft_consensus.cc:348] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:53.120922 17301 raft_consensus.cc:729] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a41c7e18d14dc0b4c5994622fe1214, State: Initialized, Role: FOLLOWER
I20260928 04:51:53.121052 17301 consensus_queue.cc:260] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8805768fa4594f3fb3e5620d1a5641a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39971 } } peers { permanent_uuid: "48a41c7e18d14dc0b4c5994622fe1214" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } peers { permanent_uuid: "905806324bdb4c0ca488ff07d4ced7ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260928 04:51:53.121337 17301 ts_tablet_manager.cc:1428] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20260928 04:51:53.121584 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:53.121486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:57523)
0928 04:51:53.121519 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:53.121557 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 04:51:53.121645 16987 tablet_copy_service.cc:342] P 905806324bdb4c0ca488ff07d4ced7ea: Request end of tablet copy session 48a41c7e18d14dc0b4c5994622fe1214-d226edd9cf754c619795a0a99e6287e4 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45019
I20260928 04:51:53.121706 16987 tablet_copy_service.cc:433] P 905806324bdb4c0ca488ff07d4ced7ea: ending tablet copy session 48a41c7e18d14dc0b4c5994622fe1214-d226edd9cf754c619795a0a99e6287e4 on tablet d226edd9cf754c619795a0a99e6287e4 with peer 48a41c7e18d14dc0b4c5994622fe1214
I20260928 04:51:53.121870 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:53.121773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:54371)
0928 04:51:53.121799 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:53.121846 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 04:51:53.481098 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:53.480915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:37665)
0928 04:51:53.481008 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:53.481045 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":39,"threads_started":1}
I20260928 04:51:53.481137 17235 raft_consensus.cc:1203] T d226edd9cf754c619795a0a99e6287e4 P 48a41c7e18d14dc0b4c5994622fe1214 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 04:51:53.481671 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:53.481541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:39265)
0928 04:51:53.481603 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:53.481639 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 04:51:53.727562 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:53.727361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:53103)
0928 04:51:53.727464 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:53.727502 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
I20260928 04:51:54.481755 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:54.481543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:36243)
0928 04:51:54.481674 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:54.481710 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260928 04:51:54.482295 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:54.482090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:33025)
0928 04:51:54.482220 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:54.482257 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":48,"threads_started":1}
I20260928 04:51:54.728258 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:54.728043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:49539)
0928 04:51:54.728168 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:54.728209 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260928 04:51:55.482396 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:55.482201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:59297)
0928 04:51:55.482316 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:55.482351 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260928 04:51:55.482977 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:55.482786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:52883)
0928 04:51:55.482910 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:55.482942 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260928 04:51:55.728895 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:55.728690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:56613)
0928 04:51:55.728815 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:55.728848 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260928 04:51:56.483182 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:56.482975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:53359)
0928 04:51:56.483105 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:56.483137 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260928 04:51:56.483505 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:56.483413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:51053)
0928 04:51:56.483458 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:56.483488 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 04:51:56.483564 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:56.483387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:50461)
0928 04:51:56.483499 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:56.483527 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260928 04:51:56.729671 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:56.729470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:59333)
0928 04:51:56.729585 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:56.729622 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260928 04:51:57.484258 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:57.484034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:37089)
0928 04:51:57.484170 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:57.484210 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260928 04:51:57.484309 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:57.484089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:43469)
0928 04:51:57.484239 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:57.484270 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260928 04:51:57.730327 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:57.730120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:35287)
0928 04:51:57.730239 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:57.730273 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260928 04:51:58.484917 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:58.484700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:49293)
0928 04:51:58.484844 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:58.484877 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260928 04:51:58.484915 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:58.484700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:47379)
0928 04:51:58.484840 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:58.484869 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260928 04:51:58.731026 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:58.730807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:57599)
0928 04:51:58.730943 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:58.730978 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260928 04:51:59.485659 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:59.485462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:48905)
0928 04:51:59.485577 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:59.485614 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260928 04:51:59.485658 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:59.485451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:54799)
0928 04:51:59.485580 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:59.485618 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260928 04:51:59.731750 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:51:59.731546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:50437)
0928 04:51:59.731673 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 04:51:59.731705 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":56,"threads_started":1}
I20260928 04:52:00.486243 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:00.486045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:44685)
0928 04:52:00.486173 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:00.486200 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260928 04:52:00.486311 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:00.486103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:43591)
0928 04:52:00.486228 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:00.486257 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260928 04:52:00.732544 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:00.732294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:56233)
0928 04:52:00.732459 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:00.732498 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260928 04:52:01.486913 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:01.486671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:47725)
0928 04:52:01.486805 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:01.486846 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260928 04:52:01.487138 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:01.486846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:59519)
0928 04:52:01.487038 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:01.487083 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260928 04:52:01.733340 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:01.733082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:47929)
0928 04:52:01.733258 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:01.733294 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":61,"threads_started":1}
I20260928 04:52:02.487731 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:02.487494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:38917)
0928 04:52:02.487624 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:02.487667 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260928 04:52:02.487906 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:02.487654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:53513)
0928 04:52:02.487793 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:02.487827 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260928 04:52:02.734009 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:02.733768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:53371)
0928 04:52:02.733918 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:02.733961 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260928 04:52:03.488560 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:03.488365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:58059)
0928 04:52:03.488486 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:03.488515 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260928 04:52:03.488680 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:03.488411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:32821)
0928 04:52:03.488571 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:03.488614 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260928 04:52:03.734691 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:03.734438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:58863)
0928 04:52:03.734605 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:03.734646 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260928 04:52:04.489318 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:04.489064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:56647)
0928 04:52:04.489207 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:04.489266 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260928 04:52:04.489346 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:04.489119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:39861)
0928 04:52:04.489270 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:04.489309 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":125,"threads_started":1}
I20260928 04:52:04.735481 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:04.735255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:37125)
0928 04:52:04.735389 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:04.735435 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260928 04:52:05.490152 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:05.489874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.1:38045)
0928 04:52:05.490042 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:05.490077 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260928 04:52:05.490172 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:05.489874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.3:55425)
0928 04:52:05.490037 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:05.490121 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260928 04:52:05.736061 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:05.735873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33349 (local address 127.0.98.2:34449)
0928 04:52:05.735983 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:05.736015 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260928 04:52:06.301767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 16848
I20260928 04:52:06.306208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 17017
I20260928 04:52:06.310700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 17151
2026-09-28T04:52:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58131 ms)
[----------] 30 tests from SecurityITest (228806 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:06 dist-test-slave-jccs krb5kdc[17349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:06 dist-test-slave-jccs krb5kdc[17349](info): set up 2 sockets
Sep 28 04:52:06 dist-test-slave-jccs krb5kdc[17349](info): commencing operation
krb5kdc: starting...
W20260928 04:52:07.348888 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:07 dist-test-slave-jccs krb5kdc[17349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:07.461566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34103
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:07.541136 17362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:07.541429 17362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:07.541468 17362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:07.542569 17362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:07.542646 17362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:07.542670 17362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:07.542690 17362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:07.542719 17362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:07.544186 17362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:07.544462 17362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:07.544697 17362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:07.549824 17368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:07.549923 17367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:07.549891 17370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:07.549906 17362 server_base.cc:1034] running on GCE node
I20260928 04:52:07.550361 17362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:07.550647 17362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:07.551801 17362 hybrid_clock.cc:648] HybridClock initialized: now 1790571127551788 us; error 30 us; skew 500 ppm
Sep 28 04:52:07 dist-test-slave-jccs krb5kdc[17349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:07.554203 17362 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:07.554625 17362 webserver.cc:460] Webserver started at http://127.0.98.62:46499/ using document root <none> and password file <none>
I20260928 04:52:07.554853 17362 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:07.554913 17362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:07.555050 17362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:07.556073 17362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "ea54870c1454476f92d1ea8597328792"
format_stamp: "Formatted at 2026-09-28 04:52:07 on dist-test-slave-jccs"
server_key: "d91f7389f1e3238b77f3e7184402f6a0"
server_key_iv: "7392edec61c6e4643effbb8d4e8c5d8f"
server_key_version: "encryptionkey@0"
I20260928 04:52:07.556427 17362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea54870c1454476f92d1ea8597328792"
format_stamp: "Formatted at 2026-09-28 04:52:07 on dist-test-slave-jccs"
server_key: "d91f7389f1e3238b77f3e7184402f6a0"
server_key_iv: "7392edec61c6e4643effbb8d4e8c5d8f"
server_key_version: "encryptionkey@0"
I20260928 04:52:07.557796 17362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 04:52:07.558462 17377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:07.558609 17362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:07.558713 17362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "ea54870c1454476f92d1ea8597328792"
format_stamp: "Formatted at 2026-09-28 04:52:07 on dist-test-slave-jccs"
server_key: "d91f7389f1e3238b77f3e7184402f6a0"
server_key_iv: "7392edec61c6e4643effbb8d4e8c5d8f"
server_key_version: "encryptionkey@0"
I20260928 04:52:07.558784 17362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:07.574419 17362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:07.576159 17362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:07.576334 17362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:07.580144 17362 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34103
I20260928 04:52:07.580175 17429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34103 every 8 connection(s)
I20260928 04:52:07.580503 17362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:07.581432 17430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:07.583541 17430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792: Bootstrap starting.
I20260928 04:52:07.584053 17430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:07.584280 17430 log.cc:824] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:07.584913 17430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792: No bootstrap required, opened a new log
I20260928 04:52:07.586141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 17362
I20260928 04:52:07.586261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:07.586450 392 external_mini_cluster.cc:1428] Setting key f33559a3dbc909a15dd9cd326e28dc8a
I20260928 04:52:07.586691 17430 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54870c1454476f92d1ea8597328792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } }
I20260928 04:52:07.586870 17430 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:07.586979 17430 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea54870c1454476f92d1ea8597328792, State: Initialized, Role: FOLLOWER
I20260928 04:52:07.587121 17430 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54870c1454476f92d1ea8597328792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } }
I20260928 04:52:07.587215 17430 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:07.587247 17430 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:07.587316 17430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:07.588444 17430 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54870c1454476f92d1ea8597328792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } }
I20260928 04:52:07.588565 17430 leader_election.cc:302] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea54870c1454476f92d1ea8597328792; no voters:
I20260928 04:52:07.588723 17430 leader_election.cc:288] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:07.588766 17435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:07.588896 17435 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [term 1 LEADER]: Becoming Leader. State: Replica: ea54870c1454476f92d1ea8597328792, State: Running, Role: LEADER
I20260928 04:52:07.588973 17430 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:07.589036 17435 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54870c1454476f92d1ea8597328792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } }
I20260928 04:52:07.589465 17437 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea54870c1454476f92d1ea8597328792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54870c1454476f92d1ea8597328792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } }
I20260928 04:52:07.589547 17437 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:07.589720 17436 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea54870c1454476f92d1ea8597328792. Latest consensus state: current_term: 1 leader_uuid: "ea54870c1454476f92d1ea8597328792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea54870c1454476f92d1ea8597328792" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34103 } } }
I20260928 04:52:07.589793 17436 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:07.589825 17444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:07.590380 17444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:07.591881 17444 catalog_manager.cc:1347] Generated new cluster ID: 1c2101b47c514012ad9b378d7fee58f2
I20260928 04:52:07.591933 17444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:52:07 dist-test-slave-jccs krb5kdc[17349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:07.598814 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:07.587008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.0.98.62:34103)
0928 04:52:07.587188 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:07.587195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:07.587241 (+ 46us) server_negotiation.cc:407] Connection header received
0928 04:52:07.587356 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:07.587363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:07.587414 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:07.587532 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:07.588354 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:07.589573 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:07.590737 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:07.591333 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:07.591435 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:07.595426 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:07.595430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:07.595431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:07.595442 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:07.597286 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:07.597680 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:07.597683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:07.597683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:07.597792 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:07.597998 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:07.597998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:07.597999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:07.598251 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:07.598385 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:07.598494 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 04:52:07.598613 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260928 04:52:07.605280 17444 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:07.605800 17444 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:07.616617 17444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792: Generated new TSK 0
I20260928 04:52:07.616808 17444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:52:07.647863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34103
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:07.746883 17456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:07.747082 17456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:07.747104 17456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:07.748236 17456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:07.748278 17456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:07.748329 17456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:52:07.749956 17456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:07.750205 17456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:07.750461 17456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:07.755440 17461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:07.755412 17464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:07.755504 17456 server_base.cc:1034] running on GCE node
W20260928 04:52:07.755407 17462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:07.755846 17456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:07.756090 17456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:07.757232 17456 hybrid_clock.cc:648] HybridClock initialized: now 1790571127757227 us; error 33 us; skew 500 ppm
Sep 28 04:52:07 dist-test-slave-jccs krb5kdc[17349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:07.759779 17456 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:52:07.760224 17456 webserver.cc:460] Webserver started at http://127.0.98.1:45375/ using document root <none> and password file <none>
I20260928 04:52:07.760434 17456 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:07.760505 17456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:07.760653 17456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:07.761677 17456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "b9e7eae8f6524ccfb744341305d523b3"
format_stamp: "Formatted at 2026-09-28 04:52:07 on dist-test-slave-jccs"
server_key: "eba03b296087be5242d2b2837db884c6"
server_key_iv: "a91bc3d5f905694b3f88f9febfe1d94c"
server_key_version: "encryptionkey@0"
I20260928 04:52:07.762015 17456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b9e7eae8f6524ccfb744341305d523b3"
format_stamp: "Formatted at 2026-09-28 04:52:07 on dist-test-slave-jccs"
server_key: "eba03b296087be5242d2b2837db884c6"
server_key_iv: "a91bc3d5f905694b3f88f9febfe1d94c"
server_key_version: "encryptionkey@0"
I20260928 04:52:07.763254 17456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:52:07.763918 17471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:07.764076 17456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:07.764170 17456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "b9e7eae8f6524ccfb744341305d523b3"
format_stamp: "Formatted at 2026-09-28 04:52:07 on dist-test-slave-jccs"
server_key: "eba03b296087be5242d2b2837db884c6"
server_key_iv: "a91bc3d5f905694b3f88f9febfe1d94c"
server_key_version: "encryptionkey@0"
I20260928 04:52:07.764246 17456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:07.785876 17456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:07.788168 17456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:07.788333 17456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:07.788571 17456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:07.788925 17456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:07.788980 17456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:07.789021 17456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:07.789048 17456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:07.795166 17456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43113
I20260928 04:52:07.795207 17584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43113 every 8 connection(s)
I20260928 04:52:07.795523 17456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:52:07 dist-test-slave-jccs krb5kdc[17349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:07.803646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 17456
I20260928 04:52:07.803752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:52:07.803987 392 external_mini_cluster.cc:1428] Setting key c18a11034aad947868f898a95792aeec
I20260928 04:52:07.804553 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:07.796226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57915 (local address 127.0.98.62:34103)
0928 04:52:07.796293 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:07.796295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:07.796502 (+ 207us) server_negotiation.cc:407] Connection header received
0928 04:52:07.797473 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:07.797474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:07.797483 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:07.797542 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:07.798097 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:07.798594 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:07.799549 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:07.799731 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:07.799781 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:07.801787 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:07.801788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:07.801788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:07.801794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:07.803156 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:07.803503 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:07.803503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:07.803504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:07.803569 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:07.803874 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:07.803875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:07.803875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:07.803978 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:07.804033 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:07.804377 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 04:52:07.804464 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:52:07.804662 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:07.796195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34103 (local address 127.0.98.1:57915)
0928 04:52:07.796374 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:07.796416 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:07.797388 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:07.797593 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:07.797596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:07.797839 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:07.798026 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:07.798029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:07.798669 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:07.798670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:07.799478 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:07.799480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:07.799798 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:07.799799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:07.799892 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:07.800364 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:07.800372 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:07.801710 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:07.803220 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:07.803225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:07.803226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:07.803434 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:07.803632 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:07.803633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:07.803633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:07.803775 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:07.804048 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:07.804055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:07.804209 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 04:52:07.804330 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 04:52:07.804459 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260928 04:52:07.805195 17585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34103
I20260928 04:52:07.805291 17585 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:07.805465 17585 heartbeater.cc:511] Master 127.0.98.62:34103 requested a full tablet report, sending...
I20260928 04:52:07.805922 17394 ts_manager.cc:194] Registered new tserver with Master: b9e7eae8f6524ccfb744341305d523b3 (127.0.98.1:43113)
I20260928 04:52:07.806474 17394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57915
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:52:07.826203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34103
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:07.903635 17590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:07.903860 17590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:07.903908 17590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:07.905041 17590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:07.905103 17590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:07.905148 17590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:52:07.906628 17590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:07.906864 17590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:07.907073 17590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:07.911880 17596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:07.911885 17598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:07.911885 17595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:07.911962 17590 server_base.cc:1034] running on GCE node
I20260928 04:52:07.912293 17590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:07.912525 17590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:07.913686 17590 hybrid_clock.cc:648] HybridClock initialized: now 1790571127913680 us; error 34 us; skew 500 ppm
Sep 28 04:52:07 dist-test-slave-jccs krb5kdc[17349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:07.916155 17590 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:52:07.916585 17590 webserver.cc:460] Webserver started at http://127.0.98.2:43631/ using document root <none> and password file <none>
I20260928 04:52:07.916800 17590 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:07.916870 17590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:07.917013 17590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:07.918033 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "434bbe2f32f541489a41e72bbbebc47e"
format_stamp: "Formatted at 2026-09-28 04:52:07 on dist-test-slave-jccs"
server_key: "1efd5723375a1dce2e2fac7937544c79"
server_key_iv: "30bf436bdc3c81f931a9a88a81021731"
server_key_version: "encryptionkey@0"
I20260928 04:52:07.918385 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "434bbe2f32f541489a41e72bbbebc47e"
format_stamp: "Formatted at 2026-09-28 04:52:07 on dist-test-slave-jccs"
server_key: "1efd5723375a1dce2e2fac7937544c79"
server_key_iv: "30bf436bdc3c81f931a9a88a81021731"
server_key_version: "encryptionkey@0"
I20260928 04:52:07.919631 17590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:52:07.920356 17605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:07.920522 17590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:07.920614 17590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "434bbe2f32f541489a41e72bbbebc47e"
format_stamp: "Formatted at 2026-09-28 04:52:07 on dist-test-slave-jccs"
server_key: "1efd5723375a1dce2e2fac7937544c79"
server_key_iv: "30bf436bdc3c81f931a9a88a81021731"
server_key_version: "encryptionkey@0"
I20260928 04:52:07.920697 17590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:07.936127 17590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:07.938179 17590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:07.938337 17590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:07.938555 17590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:07.938848 17590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:07.938906 17590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:07.938946 17590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:07.938972 17590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:07.944594 17590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38441
I20260928 04:52:07.944645 17718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38441 every 8 connection(s)
I20260928 04:52:07.944967 17590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 04:52:07 dist-test-slave-jccs krb5kdc[17349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:07.950875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 17590
I20260928 04:52:07.950989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:52:07.951220 392 external_mini_cluster.cc:1428] Setting key 34d77d091d7037e4040586531d7e6653
I20260928 04:52:07.954124 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:07.945709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56321 (local address 127.0.98.62:34103)
0928 04:52:07.945776 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:07.945778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:07.945980 (+ 202us) server_negotiation.cc:407] Connection header received
0928 04:52:07.946852 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:07.946853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:07.946862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:07.946917 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:07.947463 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:07.947925 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:07.948820 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:07.948993 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:07.949041 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:07.951147 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:07.951148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:07.951149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:07.951155 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:07.952693 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:07.953105 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:07.953106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:07.953107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:07.953164 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:07.953478 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:07.953479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:07.953480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:07.953598 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:07.953674 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:07.953942 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 04:52:07.954027 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:52:07.954341 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:07.945690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34103 (local address 127.0.98.2:56321)
0928 04:52:07.945847 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:07.945894 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:07.946764 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:07.946969 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:07.946972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:07.947214 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:07.947398 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:07.947402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:07.948003 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:07.948004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:07.948750 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:07.948751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:07.949060 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:07.949061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:07.949158 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:07.949638 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:07.949646 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:07.951049 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:07.952793 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:07.952801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:07.952803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:07.953028 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:07.953264 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:07.953265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:07.953265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:07.953378 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:07.953665 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:07.953671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:07.953813 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 04:52:07.953924 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 04:52:07.954092 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260928 04:52:07.954912 17719 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34103
I20260928 04:52:07.955029 17719 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:07.955204 17719 heartbeater.cc:511] Master 127.0.98.62:34103 requested a full tablet report, sending...
I20260928 04:52:07.955578 17394 ts_manager.cc:194] Registered new tserver with Master: 434bbe2f32f541489a41e72bbbebc47e (127.0.98.2:38441)
I20260928 04:52:07.955950 17394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56321
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:52:07.975440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34103
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:08.054930 17724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:08.055120 17724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:08.055143 17724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:08.056243 17724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:08.056285 17724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:08.056346 17724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:52:08.057837 17724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:08.058068 17724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:08.058288 17724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:08.063082 17729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:08.063089 17730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:08.063067 17732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:08.063431 17724 server_base.cc:1034] running on GCE node
I20260928 04:52:08.063660 17724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:08.063884 17724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:08.065018 17724 hybrid_clock.cc:648] HybridClock initialized: now 1790571128065009 us; error 29 us; skew 500 ppm
Sep 28 04:52:08 dist-test-slave-jccs krb5kdc[17349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:08.067396 17724 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:52:08.067850 17724 webserver.cc:460] Webserver started at http://127.0.98.3:36555/ using document root <none> and password file <none>
I20260928 04:52:08.068092 17724 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:08.068157 17724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:08.068316 17724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:08.069406 17724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "cf348cd755e74131b5131295af6b43ec"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "7fa61b74d72c2605af150bb2eec0f1af"
server_key_iv: "eb9eb81026de0c2eb95eafa44a726736"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.069769 17724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cf348cd755e74131b5131295af6b43ec"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "7fa61b74d72c2605af150bb2eec0f1af"
server_key_iv: "eb9eb81026de0c2eb95eafa44a726736"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.070994 17724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:52:08.071668 17739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.071831 17724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.071921 17724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "cf348cd755e74131b5131295af6b43ec"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "7fa61b74d72c2605af150bb2eec0f1af"
server_key_iv: "eb9eb81026de0c2eb95eafa44a726736"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.072002 17724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:08.102996 17724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:08.105407 17724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:08.105556 17724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:08.105777 17724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:08.106067 17724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:08.106145 17724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.106186 17724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:08.106215 17724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.111918 17724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43161
I20260928 04:52:08.111959 17852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43161 every 8 connection(s)
I20260928 04:52:08.112318 17724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:52:08 dist-test-slave-jccs krb5kdc[17349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:08.120057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 17724
I20260928 04:52:08.120173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:52:08.120414 392 external_mini_cluster.cc:1428] Setting key 558c315efd060c2f853f2198c4eadb85
I20260928 04:52:08.121263 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.113006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44701 (local address 127.0.98.62:34103)
0928 04:52:08.113086 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.113089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.113308 (+ 219us) server_negotiation.cc:407] Connection header received
0928 04:52:08.114260 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.114261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.114274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.114319 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:08.114940 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.115420 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.116442 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.116674 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.116749 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.118684 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:08.118685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:08.118686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:08.118691 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:08.119778 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:08.120185 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:08.120187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:08.120187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:08.120271 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:08.120573 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:08.120574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:08.120575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:08.120715 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:08.120788 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.121064 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.121152 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:52:08.121438 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.112967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34103 (local address 127.0.98.3:44701)
0928 04:52:08.113134 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.113212 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.114148 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.114388 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.114392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.114654 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:08.114846 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.114850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.115507 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.115508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.116319 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.116322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.116762 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.116763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.116935 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.117406 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:08.117414 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:08.118604 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:08.119869 (+ 1265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:08.119875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:08.119876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:08.120071 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:08.120353 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:08.120355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:08.120355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:08.120502 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:08.120792 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:08.120799 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:08.120932 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.121030 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.121200 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260928 04:52:08.121901 17853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34103
I20260928 04:52:08.122009 17853 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:08.122174 17853 heartbeater.cc:511] Master 127.0.98.62:34103 requested a full tablet report, sending...
I20260928 04:52:08.122525 17394 ts_manager.cc:194] Registered new tserver with Master: cf348cd755e74131b5131295af6b43ec (127.0.98.3:43161)
I20260928 04:52:08.122885 17394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44701
I20260928 04:52:08.132064 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:52:08.139072 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.133582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.0.98.62:34103)
0928 04:52:08.133650 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.133652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.133752 (+ 100us) server_negotiation.cc:407] Connection header received
0928 04:52:08.133815 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.133816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.133824 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.133886 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:08.134405 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.134872 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.135531 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.135697 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.135744 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.136828 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:08.136830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:08.136830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:08.136838 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:08.138196 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:08.138482 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:08.138483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:08.138483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:08.138528 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:08.138678 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:08.138678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:08.138679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:08.138805 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:08.138875 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.138961 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.139011 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:52:08.140501 17394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:52:08.142083 17394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:52:08.148624 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.144477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38441 (local address 127.0.0.1:51308)
0928 04:52:08.144723 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.144732 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.144778 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.145111 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.145113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.145116 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:08.145206 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.145208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.146555 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.146556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.147532 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.147534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.148383 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.148383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.148437 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.148448 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.148499 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.148534 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":31,"thread_start_us":92,"threads_started":1}
I20260928 04:52:08.148648 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.144731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.0.98.2:38441)
0928 04:52:08.144946 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.144951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.144961 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:52:08.144999 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.145001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.145016 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.145109 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:08.145291 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.146429 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.147612 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.148301 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.148380 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.148439 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.148522 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.148564 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
I20260928 04:52:08.148886 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.144475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43113 (local address 127.0.0.1:45212)
0928 04:52:08.144654 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.144693 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.144778 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.145411 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.145412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.145414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:08.145469 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.145483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.146454 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.146455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.147770 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.147772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.148625 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.148626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.148686 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.148690 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.148744 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.148811 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":15,"thread_start_us":62,"threads_started":1}
I20260928 04:52:08.149101 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.144573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45212 (local address 127.0.98.1:43113)
0928 04:52:08.145030 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.145034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.145043 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:52:08.145079 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.145081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.145096 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.145169 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:08.145626 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.146354 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.147855 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.148544 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.148610 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.148658 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.148986 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.149030 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":74,"threads_started":1}
I20260928 04:52:08.149628 17653 tablet_service.cc:1467] Processing CreateTablet for tablet eef08ad14c1549a0b74db03851f52a75 (DEFAULT_TABLE table=test-table [id=36c7a6ed8bc4453c984f46f641db0901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:08.149703 17519 tablet_service.cc:1467] Processing CreateTablet for tablet eef08ad14c1549a0b74db03851f52a75 (DEFAULT_TABLE table=test-table [id=36c7a6ed8bc4453c984f46f641db0901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:08.150009 17519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef08ad14c1549a0b74db03851f52a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:08.150009 17653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef08ad14c1549a0b74db03851f52a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:08.150187 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.144484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43161 (local address 127.0.0.1:43244)
0928 04:52:08.144882 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.144890 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.144929 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.145237 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.145238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.145240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:08.145284 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.145285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.146505 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.146506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.147752 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.147755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.149950 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.149951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.150014 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.150019 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.150070 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.150109 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":67,"thread_start_us":84,"threads_started":1}
I20260928 04:52:08.150432 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.144727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.0.98.3:43161)
0928 04:52:08.144902 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.144907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.144920 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:52:08.145023 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.145025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.145043 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.145134 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:08.145370 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.146414 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.148696 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.149795 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.149865 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.149915 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.150297 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.150340 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260928 04:52:08.152798 17870 tablet_bootstrap.cc:492] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: Bootstrap starting.
I20260928 04:52:08.152755 17787 tablet_service.cc:1467] Processing CreateTablet for tablet eef08ad14c1549a0b74db03851f52a75 (DEFAULT_TABLE table=test-table [id=36c7a6ed8bc4453c984f46f641db0901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:08.153039 17787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eef08ad14c1549a0b74db03851f52a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:08.153478 17870 tablet_bootstrap.cc:654] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:08.153961 17870 log.cc:824] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:08.154633 17870 tablet_bootstrap.cc:492] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: No bootstrap required, opened a new log
I20260928 04:52:08.154687 17870 ts_tablet_manager.cc:1397] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:52:08.155789 17872 tablet_bootstrap.cc:492] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: Bootstrap starting.
I20260928 04:52:08.156100 17873 tablet_bootstrap.cc:492] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3: Bootstrap starting.
I20260928 04:52:08.156550 17872 tablet_bootstrap.cc:654] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:08.156711 17873 tablet_bootstrap.cc:654] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:08.156738 17870 raft_consensus.cc:348] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 } } peers { permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 } } peers { permanent_uuid: "cf348cd755e74131b5131295af6b43ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43161 } }
I20260928 04:52:08.156966 17870 raft_consensus.cc:374] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:08.157042 17870 raft_consensus.cc:729] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 434bbe2f32f541489a41e72bbbebc47e, State: Initialized, Role: FOLLOWER
I20260928 04:52:08.157151 17872 log.cc:824] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:08.157214 17870 consensus_queue.cc:260] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 } } peers { permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 } } peers { permanent_uuid: "cf348cd755e74131b5131295af6b43ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43161 } }
I20260928 04:52:08.157326 17873 log.cc:824] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:08.157500 17870 ts_tablet_manager.cc:1428] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:52:08.157536 17719 heartbeater.cc:503] Master 127.0.98.62:34103 was elected leader, sending a full tablet report...
I20260928 04:52:08.158072 17873 tablet_bootstrap.cc:492] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3: No bootstrap required, opened a new log
I20260928 04:52:08.158102 17872 tablet_bootstrap.cc:492] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: No bootstrap required, opened a new log
I20260928 04:52:08.158172 17872 ts_tablet_manager.cc:1397] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:52:08.158181 17873 ts_tablet_manager.cc:1397] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:52:08.159807 17872 raft_consensus.cc:348] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 } } peers { permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 } } peers { permanent_uuid: "cf348cd755e74131b5131295af6b43ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43161 } }
I20260928 04:52:08.159848 17873 raft_consensus.cc:348] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 } } peers { permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 } } peers { permanent_uuid: "cf348cd755e74131b5131295af6b43ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43161 } }
I20260928 04:52:08.159961 17872 raft_consensus.cc:374] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:08.159971 17873 raft_consensus.cc:374] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:08.160009 17872 raft_consensus.cc:729] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf348cd755e74131b5131295af6b43ec, State: Initialized, Role: FOLLOWER
I20260928 04:52:08.160038 17873 raft_consensus.cc:729] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e7eae8f6524ccfb744341305d523b3, State: Initialized, Role: FOLLOWER
I20260928 04:52:08.160144 17873 consensus_queue.cc:260] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 } } peers { permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 } } peers { permanent_uuid: "cf348cd755e74131b5131295af6b43ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43161 } }
I20260928 04:52:08.160140 17872 consensus_queue.cc:260] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 } } peers { permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 } } peers { permanent_uuid: "cf348cd755e74131b5131295af6b43ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43161 } }
I20260928 04:52:08.160432 17872 ts_tablet_manager.cc:1428] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:52:08.160439 17873 ts_tablet_manager.cc:1428] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:52:08.160452 17585 heartbeater.cc:503] Master 127.0.98.62:34103 was elected leader, sending a full tablet report...
I20260928 04:52:08.160516 17853 heartbeater.cc:503] Master 127.0.98.62:34103 was elected leader, sending a full tablet report...
W20260928 04:52:08.195485 17720 tablet.cc:2369] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:52:08.202709 17877 raft_consensus.cc:482] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:52:08.202865 17877 raft_consensus.cc:504] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 } } peers { permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 } } peers { permanent_uuid: "cf348cd755e74131b5131295af6b43ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43161 } }
I20260928 04:52:08.203259 17877 leader_election.cc:288] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 434bbe2f32f541489a41e72bbbebc47e (127.0.98.2:38441), b9e7eae8f6524ccfb744341305d523b3 (127.0.98.1:43113)
I20260928 04:52:08.206499 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.203510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59145 (local address 127.0.98.2:38441)
0928 04:52:08.203555 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.203557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.203568 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:52:08.203596 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.203598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.203610 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.203668 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:08.203866 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.204617 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.205755 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.206124 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.206180 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.206188 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.206379 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.206428 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:52:08.206501 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.203425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38441 (local address 127.0.98.3:59145)
0928 04:52:08.203477 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.203488 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.203529 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.203696 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.203698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.203699 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:08.203773 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.203775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.204688 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.204689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.205660 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.205663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.206209 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.206210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.206284 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.206287 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.206355 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.206384 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":87}
I20260928 04:52:08.206560 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.203528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54135 (local address 127.0.98.1:43113)
0928 04:52:08.203571 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.203574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.203709 (+ 135us) server_negotiation.cc:407] Connection header received
0928 04:52:08.203747 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.203748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.203757 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.203799 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:08.203950 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.204632 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.205778 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.206152 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.206238 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.206248 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.206438 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.206487 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:52:08.206581 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.203467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43113 (local address 127.0.98.3:54135)
0928 04:52:08.203631 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.203639 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.203692 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.203827 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.203828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.203830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:08.203884 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.203885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.204718 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.204719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.205671 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.205674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.206267 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.206268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.206358 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.206362 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.206418 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.206450 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260928 04:52:08.206823 17673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef08ad14c1549a0b74db03851f52a75" candidate_uuid: "cf348cd755e74131b5131295af6b43ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "434bbe2f32f541489a41e72bbbebc47e" is_pre_election: true
I20260928 04:52:08.206869 17539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef08ad14c1549a0b74db03851f52a75" candidate_uuid: "cf348cd755e74131b5131295af6b43ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e7eae8f6524ccfb744341305d523b3" is_pre_election: true
I20260928 04:52:08.206979 17539 raft_consensus.cc:2454] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf348cd755e74131b5131295af6b43ec in term 0.
I20260928 04:52:08.206979 17673 raft_consensus.cc:2454] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf348cd755e74131b5131295af6b43ec in term 0.
I20260928 04:52:08.207193 17741 leader_election.cc:302] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 434bbe2f32f541489a41e72bbbebc47e, cf348cd755e74131b5131295af6b43ec; no voters:
I20260928 04:52:08.207332 17877 raft_consensus.cc:2781] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:52:08.207382 17877 raft_consensus.cc:482] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:52:08.207417 17877 raft_consensus.cc:3037] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:08.208052 17877 raft_consensus.cc:504] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 } } peers { permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 } } peers { permanent_uuid: "cf348cd755e74131b5131295af6b43ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43161 } }
I20260928 04:52:08.208179 17877 leader_election.cc:288] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [CANDIDATE]: Term 1 election: Requested vote from peers 434bbe2f32f541489a41e72bbbebc47e (127.0.98.2:38441), b9e7eae8f6524ccfb744341305d523b3 (127.0.98.1:43113)
I20260928 04:52:08.208346 17539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef08ad14c1549a0b74db03851f52a75" candidate_uuid: "cf348cd755e74131b5131295af6b43ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e7eae8f6524ccfb744341305d523b3"
I20260928 04:52:08.208415 17539 raft_consensus.cc:3037] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:08.209105 17539 raft_consensus.cc:2454] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf348cd755e74131b5131295af6b43ec in term 1.
I20260928 04:52:08.209267 17673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef08ad14c1549a0b74db03851f52a75" candidate_uuid: "cf348cd755e74131b5131295af6b43ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "434bbe2f32f541489a41e72bbbebc47e"
I20260928 04:52:08.209297 17743 leader_election.cc:302] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9e7eae8f6524ccfb744341305d523b3, cf348cd755e74131b5131295af6b43ec; no voters:
I20260928 04:52:08.209352 17673 raft_consensus.cc:3037] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:08.209391 17877 raft_consensus.cc:2781] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:08.209534 17877 raft_consensus.cc:686] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 1 LEADER]: Becoming Leader. State: Replica: cf348cd755e74131b5131295af6b43ec, State: Running, Role: LEADER
I20260928 04:52:08.209631 17877 consensus_queue.cc:237] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 } } peers { permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 } } peers { permanent_uuid: "cf348cd755e74131b5131295af6b43ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43161 } }
I20260928 04:52:08.210255 17673 raft_consensus.cc:2454] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf348cd755e74131b5131295af6b43ec in term 1.
I20260928 04:52:08.210409 17393 catalog_manager.cc:5495] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec reported cstate change: term changed from 0 to 1, leader changed from <none> to cf348cd755e74131b5131295af6b43ec (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cf348cd755e74131b5131295af6b43ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf348cd755e74131b5131295af6b43ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43161 } health_report { overall_health: HEALTHY } } }
I20260928 04:52:08.224332 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.221906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.0.98.3:43161)
0928 04:52:08.221958 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.221960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.221970 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:52:08.222041 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.222042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.222062 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.222136 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:08.222285 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.222770 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.223559 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.223753 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.223827 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.223998 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:08.224093 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:08.224182 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.224207 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.224227 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 04:52:08.226045 17673 raft_consensus.cc:1261] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e [term 1 FOLLOWER]: Refusing update from remote peer cf348cd755e74131b5131295af6b43ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:08.226056 17539 raft_consensus.cc:1261] T eef08ad14c1549a0b74db03851f52a75 P b9e7eae8f6524ccfb744341305d523b3 [term 1 FOLLOWER]: Refusing update from remote peer cf348cd755e74131b5131295af6b43ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:08.226373 17880 consensus_queue.cc:1035] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "434bbe2f32f541489a41e72bbbebc47e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:08.226507 17877 consensus_queue.cc:1035] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9e7eae8f6524ccfb744341305d523b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:08.229677 17885 mvcc.cc:205] Tried to move back new op lower bound from 7334179341211619328 to 7334179341147635712. Current Snapshot: MvccSnapshot[applied={T|T < 7334179341211619328}]
I20260928 04:52:08.230775 17393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:08.230876 17393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:08.231618 17393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea54870c1454476f92d1ea8597328792: Sending DeleteTablet for 3 replicas of tablet eef08ad14c1549a0b74db03851f52a75
I20260928 04:52:08.231922 17787 tablet_service.cc:1514] Processing DeleteTablet for tablet eef08ad14c1549a0b74db03851f52a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43244
I20260928 04:52:08.232018 17653 tablet_service.cc:1514] Processing DeleteTablet for tablet eef08ad14c1549a0b74db03851f52a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51308
I20260928 04:52:08.232112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 17456
I20260928 04:52:08.232108 17519 tablet_service.cc:1514] Processing DeleteTablet for tablet eef08ad14c1549a0b74db03851f52a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45212
I20260928 04:52:08.232228 17893 tablet_replica.cc:330] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: stopping tablet replica
I20260928 04:52:08.232340 17893 raft_consensus.cc:2229] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:52:08.232448 17893 raft_consensus.cc:2258] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:08.233093 17893 ts_tablet_manager.cc:1905] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:08.235004 17893 ts_tablet_manager.cc:1918] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:08.235082 17893 log.cc:1196] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/eef08ad14c1549a0b74db03851f52a75
I20260928 04:52:08.235380 17893 ts_tablet_manager.cc:1939] T eef08ad14c1549a0b74db03851f52a75 P 434bbe2f32f541489a41e72bbbebc47e: Deleting consensus metadata
I20260928 04:52:08.235791 17379 catalog_manager.cc:4841] TS 434bbe2f32f541489a41e72bbbebc47e (127.0.98.2:38441): tablet eef08ad14c1549a0b74db03851f52a75 (table test-table [id=36c7a6ed8bc4453c984f46f641db0901]) successfully deleted
W20260928 04:52:08.238255 17381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43113 (error 108)
I20260928 04:52:08.238459 17892 tablet_replica.cc:330] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: stopping tablet replica
I20260928 04:52:08.238569 17892 raft_consensus.cc:2229] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:52:08.238664 17892 raft_consensus.cc:2258] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:08.238735 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.238634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43113 (local address 127.0.0.1:45214)
0928 04:52:08.238675 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.238704 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 04:52:08.238824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 17590
W20260928 04:52:08.238855 17381 catalog_manager.cc:4584] TS b9e7eae8f6524ccfb744341305d523b3 (127.0.98.1:43113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eef08ad14c1549a0b74db03851f52a75: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43113: connect: Connection refused (error 111)
I20260928 04:52:08.239620 17892 ts_tablet_manager.cc:1905] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:08.241565 17892 ts_tablet_manager.cc:1918] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:08.241627 17892 log.cc:1196] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/eef08ad14c1549a0b74db03851f52a75
I20260928 04:52:08.241888 17892 ts_tablet_manager.cc:1939] T eef08ad14c1549a0b74db03851f52a75 P cf348cd755e74131b5131295af6b43ec: Deleting consensus metadata
I20260928 04:52:08.242354 17378 catalog_manager.cc:4841] TS cf348cd755e74131b5131295af6b43ec (127.0.98.3:43161): tablet eef08ad14c1549a0b74db03851f52a75 (table test-table [id=36c7a6ed8bc4453c984f46f641db0901]) successfully deleted
I20260928 04:52:08.245191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 17724
I20260928 04:52:08.249972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 17362
2026-09-28T04:52:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1938 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T04:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:08Z Disabled control of system clock
I20260928 04:52:08.276935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41161
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:52:08.357873 17898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:08.358103 17898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:08.358124 17898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:08.359365 17898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:08.359411 17898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:08.359424 17898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:08.359437 17898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:08.359449 17898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:08.360800 17898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:08.361001 17898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:08.361315 17898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:08.365859 17903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:08.365870 17904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:08.365866 17906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:08.366097 17898 server_base.cc:1034] running on GCE node
I20260928 04:52:08.366350 17898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:08.366606 17898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:08.367759 17898 hybrid_clock.cc:648] HybridClock initialized: now 1790571128367751 us; error 37 us; skew 500 ppm
I20260928 04:52:08.369024 17898 webserver.cc:460] Webserver started at http://127.0.98.62:39727/ using document root <none> and password file <none>
I20260928 04:52:08.369323 17898 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:08.369392 17898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:08.369560 17898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:08.370632 17898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "184caaca529d498484c7822a8a6c50c4"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "f78b2c7429e725ba24e2b222d14be593"
server_key_iv: "8343df62943b2fba8e92d79c5c2f377c"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.370985 17898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "184caaca529d498484c7822a8a6c50c4"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "f78b2c7429e725ba24e2b222d14be593"
server_key_iv: "8343df62943b2fba8e92d79c5c2f377c"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.372216 17898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:08.372920 17912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.373081 17898 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:08.373174 17898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "184caaca529d498484c7822a8a6c50c4"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "f78b2c7429e725ba24e2b222d14be593"
server_key_iv: "8343df62943b2fba8e92d79c5c2f377c"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.373329 17898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:08.402436 17898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:08.402771 17898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:08.402925 17898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:08.406983 17898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41161
I20260928 04:52:08.407024 17964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41161 every 8 connection(s)
I20260928 04:52:08.407390 17898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:08.408306 17965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:08.410570 17965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4: Bootstrap starting.
I20260928 04:52:08.411190 17965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:08.411461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 17898
I20260928 04:52:08.411533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:08.411518 17965 log.cc:824] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:08.411758 392 external_mini_cluster.cc:1428] Setting key dda1065e03cd0f900ec89808fb61cfb9
I20260928 04:52:08.412225 17965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4: No bootstrap required, opened a new log
I20260928 04:52:08.413997 17965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184caaca529d498484c7822a8a6c50c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41161 } }
I20260928 04:52:08.414156 17965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:08.414220 17965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184caaca529d498484c7822a8a6c50c4, State: Initialized, Role: FOLLOWER
I20260928 04:52:08.414393 17965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184caaca529d498484c7822a8a6c50c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41161 } }
I20260928 04:52:08.414498 17965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:08.414529 17965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:08.414573 17965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:08.415309 17965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184caaca529d498484c7822a8a6c50c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41161 } }
I20260928 04:52:08.415414 17965 leader_election.cc:302] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 184caaca529d498484c7822a8a6c50c4; no voters:
I20260928 04:52:08.415565 17965 leader_election.cc:288] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:08.415629 17970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:08.415805 17965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:08.415808 17970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [term 1 LEADER]: Becoming Leader. State: Replica: 184caaca529d498484c7822a8a6c50c4, State: Running, Role: LEADER
I20260928 04:52:08.415992 17970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184caaca529d498484c7822a8a6c50c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41161 } }
I20260928 04:52:08.416445 17972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184caaca529d498484c7822a8a6c50c4. Latest consensus state: current_term: 1 leader_uuid: "184caaca529d498484c7822a8a6c50c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184caaca529d498484c7822a8a6c50c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41161 } } }
I20260928 04:52:08.416522 17972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:08.416744 17971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "184caaca529d498484c7822a8a6c50c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184caaca529d498484c7822a8a6c50c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41161 } } }
I20260928 04:52:08.416885 17971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:08.417097 17979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:08.417557 17979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:08.418987 17979 catalog_manager.cc:1347] Generated new cluster ID: 31ac75b9c7e04006ab4ba4ddf40b8778
I20260928 04:52:08.419037 17979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:08.419667 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.412301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.0.98.62:41161)
0928 04:52:08.412527 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.412534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.412556 (+ 22us) server_negotiation.cc:407] Connection header received
0928 04:52:08.413982 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.413989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.414048 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.414189 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:08.414447 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.416362 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.417491 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.418130 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.418239 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.418996 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:08.419000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:08.419000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:08.419189 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:08.419230 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:08.419241 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:08.419312 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.419405 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.419478 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260928 04:52:08.444121 17979 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:08.444666 17979 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:08.449402 17979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4: Generated new TSK 0
I20260928 04:52:08.449602 17979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:08.457530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41161
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:52:08.554394 17989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:08.554592 17989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:08.554612 17989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:08.555714 17989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:08.555752 17989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:08.555776 17989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:52:08.557054 17989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:08.557314 17989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:08.557626 17989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:08.563213 17995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:08.563221 17994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:08.563436 17989 server_base.cc:1034] running on GCE node
W20260928 04:52:08.563552 17997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:08.563738 17989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:08.563987 17989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:08.565165 17989 hybrid_clock.cc:648] HybridClock initialized: now 1790571128565150 us; error 28 us; skew 500 ppm
I20260928 04:52:08.566623 17989 webserver.cc:460] Webserver started at http://127.0.98.1:34607/ using document root <none> and password file <none>
I20260928 04:52:08.566870 17989 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:08.566924 17989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:08.567052 17989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:08.568264 17989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "deadeaaf8f0845d4b4c596c6fa087568"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "0f2f6cfbc804f7d14bf93eba947d171c"
server_key_iv: "f6db945118ab9409c3b9c0efd8f68881"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.568624 17989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "deadeaaf8f0845d4b4c596c6fa087568"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "0f2f6cfbc804f7d14bf93eba947d171c"
server_key_iv: "f6db945118ab9409c3b9c0efd8f68881"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.570091 17989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:52:08.570746 18003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.570909 17989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.570973 17989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "deadeaaf8f0845d4b4c596c6fa087568"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "0f2f6cfbc804f7d14bf93eba947d171c"
server_key_iv: "f6db945118ab9409c3b9c0efd8f68881"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.571045 17989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:08.586467 17989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:08.586720 17989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:08.586817 17989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:08.586994 17989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:08.587239 17989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:08.587267 17989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.587288 17989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:08.587301 17989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.592914 17989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35627
I20260928 04:52:08.593036 18116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35627 every 8 connection(s)
I20260928 04:52:08.593381 17989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:52:08.596094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 17989
I20260928 04:52:08.596170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:52:08.596370 392 external_mini_cluster.cc:1428] Setting key 250546d1e22eddfb61d31490be573d36
I20260928 04:52:08.597432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41161
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:52:08.599058 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.594131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51299 (local address 127.0.98.62:41161)
0928 04:52:08.594215 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.594217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.594411 (+ 194us) server_negotiation.cc:407] Connection header received
0928 04:52:08.594868 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.594869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.594882 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.594935 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:08.595579 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.596152 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.597570 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.597883 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.597957 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.598804 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:08.598806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:08.598806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:08.598810 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:08.598823 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:08.598827 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:08.598872 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.598975 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.599001 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 04:52:08.599318 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.594084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41161 (local address 127.0.98.1:51299)
0928 04:52:08.594295 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.594329 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.594788 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.595020 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.595024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.595032 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:08.595510 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.595514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.596294 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.596295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.597455 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.597458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.597970 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.597971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.598073 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.598637 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:52:08.598638 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:08.598680 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:52:08.598711 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:08.598891 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:08.598897 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:08.598899 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.598969 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.599078 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260928 04:52:08.599731 18117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41161
I20260928 04:52:08.599838 18117 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:08.600013 18117 heartbeater.cc:511] Master 127.0.98.62:41161 requested a full tablet report, sending...
I20260928 04:52:08.600508 17929 ts_manager.cc:194] Registered new tserver with Master: deadeaaf8f0845d4b4c596c6fa087568 (127.0.98.1:35627)
I20260928 04:52:08.601157 17929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51299
W20260928 04:52:08.677613 18120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:08.677845 18120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:08.677879 18120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:08.679059 18120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:08.679140 18120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:08.679177 18120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:52:08.680488 18120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:08.680744 18120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:08.681030 18120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:08.685905 18126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:08.685889 18128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:08.685912 18125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:08.685950 18120 server_base.cc:1034] running on GCE node
I20260928 04:52:08.686309 18120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:08.686548 18120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:08.687703 18120 hybrid_clock.cc:648] HybridClock initialized: now 1790571128687695 us; error 38 us; skew 500 ppm
I20260928 04:52:08.688969 18120 webserver.cc:460] Webserver started at http://127.0.98.2:42779/ using document root <none> and password file <none>
I20260928 04:52:08.689268 18120 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:08.689335 18120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:08.689471 18120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:08.690493 18120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "eea974524fd44b0bb2deced006a400eb"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "710add3c24312daba15c21aa76da1337"
server_key_iv: "fbb471dd8d350874c21a3b919e4b74af"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.690837 18120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eea974524fd44b0bb2deced006a400eb"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "710add3c24312daba15c21aa76da1337"
server_key_iv: "fbb471dd8d350874c21a3b919e4b74af"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.692173 18120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:08.692905 18134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.693095 18120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:08.693216 18120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "eea974524fd44b0bb2deced006a400eb"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "710add3c24312daba15c21aa76da1337"
server_key_iv: "fbb471dd8d350874c21a3b919e4b74af"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.693307 18120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:08.713954 18120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:08.714290 18120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:08.714459 18120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:08.714695 18120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:08.715023 18120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:08.715086 18120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.715132 18120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:08.715161 18120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.721001 18120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46325
I20260928 04:52:08.721064 18247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46325 every 8 connection(s)
I20260928 04:52:08.721437 18120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:52:08.722553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 18120
I20260928 04:52:08.722640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:52:08.722847 392 external_mini_cluster.cc:1428] Setting key 5b20f7160e1b07818b760b805cf0391d
I20260928 04:52:08.723831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41161
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:52:08.727048 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.722161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36869 (local address 127.0.98.62:41161)
0928 04:52:08.722213 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.722216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.722489 (+ 273us) server_negotiation.cc:407] Connection header received
0928 04:52:08.723025 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.723026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.723038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.723117 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:08.723846 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.724622 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.725623 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.725831 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.725915 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.726774 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:08.726775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:08.726775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:08.726780 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:08.726795 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:08.726799 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:08.726845 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.726947 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.726976 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:52:08.727270 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.722160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41161 (local address 127.0.98.2:36869)
0928 04:52:08.722361 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.722395 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.722941 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.723187 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.723190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.723198 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:08.723755 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.723760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.724703 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.724704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.725532 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.725534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.725923 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.725924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.726048 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.726639 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:52:08.726640 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:08.726666 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:52:08.726686 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:08.726844 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:08.726849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:08.726852 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.726947 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.727038 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260928 04:52:08.727772 18248 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41161
I20260928 04:52:08.727871 18248 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:08.728081 18248 heartbeater.cc:511] Master 127.0.98.62:41161 requested a full tablet report, sending...
I20260928 04:52:08.728438 17929 ts_manager.cc:194] Registered new tserver with Master: eea974524fd44b0bb2deced006a400eb (127.0.98.2:46325)
I20260928 04:52:08.728971 17929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36869
W20260928 04:52:08.803288 18251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:08.803476 18251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:08.803496 18251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:08.804553 18251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:08.804584 18251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:08.804607 18251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:52:08.805882 18251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:08.806062 18251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:08.806277 18251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:08.811038 18259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:08.811012 18256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:08.811024 18257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:08.811331 18251 server_base.cc:1034] running on GCE node
I20260928 04:52:08.811508 18251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:08.811690 18251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:08.812825 18251 hybrid_clock.cc:648] HybridClock initialized: now 1790571128812817 us; error 32 us; skew 500 ppm
I20260928 04:52:08.814019 18251 webserver.cc:460] Webserver started at http://127.0.98.3:32873/ using document root <none> and password file <none>
I20260928 04:52:08.814256 18251 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:08.814313 18251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:08.814455 18251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:08.815487 18251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "83048b3bcd13469a9d870986d6e53138"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "d703116ba752c18d47a10276271d39cb"
server_key_iv: "fe1f3e16b38ae826ed0ed6fcd98beb49"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.815866 18251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "83048b3bcd13469a9d870986d6e53138"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "d703116ba752c18d47a10276271d39cb"
server_key_iv: "fe1f3e16b38ae826ed0ed6fcd98beb49"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.817284 18251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:52:08.817996 18265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.818177 18251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:08.818243 18251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "83048b3bcd13469a9d870986d6e53138"
format_stamp: "Formatted at 2026-09-28 04:52:08 on dist-test-slave-jccs"
server_key: "d703116ba752c18d47a10276271d39cb"
server_key_iv: "fe1f3e16b38ae826ed0ed6fcd98beb49"
server_key_version: "encryptionkey@0"
I20260928 04:52:08.818289 18251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:08.840948 18251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:08.841243 18251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:08.841344 18251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:08.841549 18251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:08.841885 18251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:08.841917 18251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.841938 18251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:08.841985 18251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:08.847437 18251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36787
I20260928 04:52:08.847517 18378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36787 every 8 connection(s)
I20260928 04:52:08.847843 18251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:52:08.848861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 18251
I20260928 04:52:08.848984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:52:08.849217 392 external_mini_cluster.cc:1428] Setting key fd293b418d78eba76d8b285c0d3713e1
I20260928 04:52:08.852948 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.848606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50221 (local address 127.0.98.62:41161)
0928 04:52:08.848676 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.848678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.848910 (+ 232us) server_negotiation.cc:407] Connection header received
0928 04:52:08.849531 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.849532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.849542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.849619 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:08.850394 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.850887 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.851807 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.851985 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.852034 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.852735 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:08.852736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:08.852737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:08.852740 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:08.852753 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:08.852757 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:08.852795 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.852871 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.852892 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 04:52:08.853156 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.848585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41161 (local address 127.0.98.3:50221)
0928 04:52:08.848775 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.848815 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.849429 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.849697 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.849701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.849712 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:08.850319 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.850323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.850979 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.850980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.851732 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.851734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.852051 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.852052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.852163 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.852621 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:52:08.852621 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:08.852646 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:52:08.852665 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:08.852805 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:08.852808 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:08.852810 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.852856 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.852922 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260928 04:52:08.853632 18379 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41161
I20260928 04:52:08.853729 18379 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:08.853897 18379 heartbeater.cc:511] Master 127.0.98.62:41161 requested a full tablet report, sending...
I20260928 04:52:08.854241 17929 ts_manager.cc:194] Registered new tserver with Master: 83048b3bcd13469a9d870986d6e53138 (127.0.98.3:36787)
I20260928 04:52:08.854557 17929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50221
I20260928 04:52:08.860934 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:52:08.865217 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.862366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.0.98.62:41161)
0928 04:52:08.862446 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.862448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.862520 (+ 72us) server_negotiation.cc:407] Connection header received
0928 04:52:08.862594 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.862595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.862606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.862658 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:08.862824 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.863304 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.864061 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.864273 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.864340 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.864944 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:08.864946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:08.864946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:08.864950 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:08.864964 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:08.864968 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:08.865045 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.865097 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.865120 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 04:52:08.866613 17929 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:52:08.868122 17929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:52:08.875312 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.870770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46325 (local address 127.0.0.1:39426)
0928 04:52:08.871109 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.871120 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.871169 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.871658 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.871658 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.871661 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:08.871707 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.871708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.872865 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.872866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.873934 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.873936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.874916 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.874917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.875030 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.875045 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.875123 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.875195 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":45,"thread_start_us":123,"threads_started":1}
I20260928 04:52:08.875311 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.870987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36787 (local address 127.0.0.1:54114)
0928 04:52:08.871430 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.871440 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.871486 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.871762 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.871763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.871765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:08.871795 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.871796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.872626 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.872628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.874114 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.874116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.874991 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.874991 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.875039 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.875048 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.875123 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.875195 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":26,"thread_start_us":70,"threads_started":1}
I20260928 04:52:08.875276 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.870956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.0.98.2:46325)
0928 04:52:08.871454 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.871459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.871469 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:52:08.871504 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.871506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.871520 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.871642 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:08.871764 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.872799 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.874032 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.874833 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.874913 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.875022 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.875146 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.875189 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":100,"threads_started":1}
I20260928 04:52:08.875752 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.871180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.0.98.3:36787)
0928 04:52:08.871319 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.871324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.871562 (+ 238us) server_negotiation.cc:407] Connection header received
0928 04:52:08.871611 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.871613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.871628 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.871700 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:08.871849 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.872528 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.874206 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.874917 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.874981 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.875054 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.875346 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.875387 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260928 04:52:08.876245 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.870869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.0.98.1:35627)
0928 04:52:08.871175 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.871180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.871190 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:52:08.871229 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.871231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.871249 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.871336 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:08.871506 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.872668 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.875115 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.875899 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.875975 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.876058 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.876121 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.876163 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20260928 04:52:08.876415 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.870768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35627 (local address 127.0.0.1:35634)
0928 04:52:08.870957 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.871002 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.871085 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.871337 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.871340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.871342 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:08.871412 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.871414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.872752 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.872753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.874107 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.874110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.875978 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.875979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.876034 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.876036 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.876106 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.876280 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":95,"thread_start_us":56,"threads_started":1}
I20260928 04:52:08.876543 18182 tablet_service.cc:1467] Processing CreateTablet for tablet 750b17cbf1674b968683ce44e3f00ad8 (DEFAULT_TABLE table=test-table [id=42c9ed8d31084488985425402b935cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:08.876683 18313 tablet_service.cc:1467] Processing CreateTablet for tablet 750b17cbf1674b968683ce44e3f00ad8 (DEFAULT_TABLE table=test-table [id=42c9ed8d31084488985425402b935cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:08.876950 18182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750b17cbf1674b968683ce44e3f00ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:08.877039 18313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750b17cbf1674b968683ce44e3f00ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:08.877010 18051 tablet_service.cc:1467] Processing CreateTablet for tablet 750b17cbf1674b968683ce44e3f00ad8 (DEFAULT_TABLE table=test-table [id=42c9ed8d31084488985425402b935cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:08.877329 18051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750b17cbf1674b968683ce44e3f00ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:08.879621 18393 tablet_bootstrap.cc:492] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: Bootstrap starting.
I20260928 04:52:08.880239 18394 tablet_bootstrap.cc:492] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: Bootstrap starting.
I20260928 04:52:08.880330 18393 tablet_bootstrap.cc:654] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:08.880389 18395 tablet_bootstrap.cc:492] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568: Bootstrap starting.
I20260928 04:52:08.880694 18393 log.cc:824] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:08.880955 18394 tablet_bootstrap.cc:654] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:08.881050 18395 tablet_bootstrap.cc:654] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:08.881325 18394 log.cc:824] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:08.881433 18395 log.cc:824] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:08.881481 18393 tablet_bootstrap.cc:492] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: No bootstrap required, opened a new log
I20260928 04:52:08.881536 18393 ts_tablet_manager.cc:1397] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:52:08.882189 18395 tablet_bootstrap.cc:492] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568: No bootstrap required, opened a new log
I20260928 04:52:08.882267 18395 ts_tablet_manager.cc:1397] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:52:08.884203 18393 raft_consensus.cc:348] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadeaaf8f0845d4b4c596c6fa087568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35627 } } peers { permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 } } peers { permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260928 04:52:08.884385 18393 raft_consensus.cc:374] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:08.884418 18393 raft_consensus.cc:729] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83048b3bcd13469a9d870986d6e53138, State: Initialized, Role: FOLLOWER
I20260928 04:52:08.884536 18393 consensus_queue.cc:260] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadeaaf8f0845d4b4c596c6fa087568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35627 } } peers { permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 } } peers { permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260928 04:52:08.884780 18393 ts_tablet_manager.cc:1428] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 04:52:08.884720 18395 raft_consensus.cc:348] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadeaaf8f0845d4b4c596c6fa087568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35627 } } peers { permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 } } peers { permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260928 04:52:08.884824 18379 heartbeater.cc:503] Master 127.0.98.62:41161 was elected leader, sending a full tablet report...
I20260928 04:52:08.884899 18395 raft_consensus.cc:374] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:08.884954 18395 raft_consensus.cc:729] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deadeaaf8f0845d4b4c596c6fa087568, State: Initialized, Role: FOLLOWER
I20260928 04:52:08.885128 18395 consensus_queue.cc:260] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadeaaf8f0845d4b4c596c6fa087568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35627 } } peers { permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 } } peers { permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260928 04:52:08.885371 18394 tablet_bootstrap.cc:492] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: No bootstrap required, opened a new log
I20260928 04:52:08.885401 18395 ts_tablet_manager.cc:1428] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:52:08.885438 18394 ts_tablet_manager.cc:1397] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 04:52:08.885476 18117 heartbeater.cc:503] Master 127.0.98.62:41161 was elected leader, sending a full tablet report...
I20260928 04:52:08.887328 18394 raft_consensus.cc:348] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadeaaf8f0845d4b4c596c6fa087568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35627 } } peers { permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 } } peers { permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260928 04:52:08.887439 18394 raft_consensus.cc:374] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:08.887458 18394 raft_consensus.cc:729] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea974524fd44b0bb2deced006a400eb, State: Initialized, Role: FOLLOWER
I20260928 04:52:08.887600 18394 consensus_queue.cc:260] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadeaaf8f0845d4b4c596c6fa087568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35627 } } peers { permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 } } peers { permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260928 04:52:08.887845 18394 ts_tablet_manager.cc:1428] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 04:52:08.887923 18248 heartbeater.cc:503] Master 127.0.98.62:41161 was elected leader, sending a full tablet report...
I20260928 04:52:08.894394 18400 raft_consensus.cc:482] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:52:08.894464 18400 raft_consensus.cc:504] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadeaaf8f0845d4b4c596c6fa087568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35627 } } peers { permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 } } peers { permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260928 04:52:08.894695 18400 leader_election.cc:288] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eea974524fd44b0bb2deced006a400eb (127.0.98.2:46325), 83048b3bcd13469a9d870986d6e53138 (127.0.98.3:36787)
I20260928 04:52:08.897332 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.894899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54239 (local address 127.0.98.3:36787)
0928 04:52:08.894991 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.894994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.895001 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:52:08.895023 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.895024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.895032 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.895080 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:08.895205 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.895634 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.896550 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.896999 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.897070 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.897078 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.897244 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.897276 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 04:52:08.897333 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.894842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36787 (local address 127.0.98.1:54239)
0928 04:52:08.894897 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.894909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.894948 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.895081 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.895081 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.895083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:08.895142 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.895144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.895717 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.895718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.896461 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.896463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.897068 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.897069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.897132 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.897135 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.897237 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.897274 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 04:52:08.897440 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.894961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50007 (local address 127.0.98.2:46325)
0928 04:52:08.895010 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.895012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.895047 (+ 35us) server_negotiation.cc:407] Connection header received
0928 04:52:08.895123 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.895124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.895132 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.895187 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:08.895331 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.895830 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.896722 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.897097 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.897171 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.897179 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.897352 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.897384 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:52:08.897449 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.894854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46325 (local address 127.0.98.1:50007)
0928 04:52:08.894987 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.894999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:08.895064 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:08.895191 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:08.895191 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:08.895193 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:08.895269 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.895271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.895907 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.895929 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.896628 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:08.896630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.897204 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.897205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:08.897267 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.897276 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:52:08.897344 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:52:08.897377 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260928 04:52:08.897621 18333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750b17cbf1674b968683ce44e3f00ad8" candidate_uuid: "deadeaaf8f0845d4b4c596c6fa087568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83048b3bcd13469a9d870986d6e53138" is_pre_election: true
I20260928 04:52:08.897713 18333 raft_consensus.cc:2454] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deadeaaf8f0845d4b4c596c6fa087568 in term 0.
I20260928 04:52:08.897887 18004 leader_election.cc:302] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83048b3bcd13469a9d870986d6e53138, deadeaaf8f0845d4b4c596c6fa087568; no voters:
I20260928 04:52:08.897986 18400 raft_consensus.cc:2781] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:52:08.898032 18400 raft_consensus.cc:482] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:52:08.898056 18400 raft_consensus.cc:3037] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:08.898404 18202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750b17cbf1674b968683ce44e3f00ad8" candidate_uuid: "deadeaaf8f0845d4b4c596c6fa087568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea974524fd44b0bb2deced006a400eb" is_pre_election: true
I20260928 04:52:08.898520 18202 raft_consensus.cc:2454] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate deadeaaf8f0845d4b4c596c6fa087568 in term 0.
I20260928 04:52:08.898774 18400 raft_consensus.cc:504] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadeaaf8f0845d4b4c596c6fa087568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35627 } } peers { permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 } } peers { permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260928 04:52:08.898897 18400 leader_election.cc:288] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [CANDIDATE]: Term 1 election: Requested vote from peers eea974524fd44b0bb2deced006a400eb (127.0.98.2:46325), 83048b3bcd13469a9d870986d6e53138 (127.0.98.3:36787)
I20260928 04:52:08.899013 18202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750b17cbf1674b968683ce44e3f00ad8" candidate_uuid: "deadeaaf8f0845d4b4c596c6fa087568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea974524fd44b0bb2deced006a400eb"
I20260928 04:52:08.899044 18333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750b17cbf1674b968683ce44e3f00ad8" candidate_uuid: "deadeaaf8f0845d4b4c596c6fa087568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83048b3bcd13469a9d870986d6e53138"
I20260928 04:52:08.899118 18333 raft_consensus.cc:3037] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:08.899084 18202 raft_consensus.cc:3037] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:08.899693 18333 raft_consensus.cc:2454] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deadeaaf8f0845d4b4c596c6fa087568 in term 1.
I20260928 04:52:08.899732 18202 raft_consensus.cc:2454] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate deadeaaf8f0845d4b4c596c6fa087568 in term 1.
I20260928 04:52:08.899887 18004 leader_election.cc:302] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83048b3bcd13469a9d870986d6e53138, deadeaaf8f0845d4b4c596c6fa087568; no voters:
I20260928 04:52:08.900013 18400 raft_consensus.cc:2781] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:08.900173 18400 raft_consensus.cc:686] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [term 1 LEADER]: Becoming Leader. State: Replica: deadeaaf8f0845d4b4c596c6fa087568, State: Running, Role: LEADER
I20260928 04:52:08.900256 18400 consensus_queue.cc:237] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadeaaf8f0845d4b4c596c6fa087568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35627 } } peers { permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 } } peers { permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } }
I20260928 04:52:08.900940 17928 catalog_manager.cc:5495] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 reported cstate change: term changed from 0 to 1, leader changed from <none> to deadeaaf8f0845d4b4c596c6fa087568 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "deadeaaf8f0845d4b4c596c6fa087568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deadeaaf8f0845d4b4c596c6fa087568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 } health_report { overall_health: UNKNOWN } } }
I20260928 04:52:08.910211 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.907875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.0.98.1:35627)
0928 04:52:08.907919 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:08.907921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:08.907930 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:52:08.907994 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:08.907995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:08.908015 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:08.908100 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:08.908208 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.908643 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.909421 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:08.909609 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:08.909682 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:08.909863 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:08.909969 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:08.910063 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:08.910083 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 04:52:08.910100 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:52:08.911860 18202 raft_consensus.cc:1261] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb [term 1 FOLLOWER]: Refusing update from remote peer deadeaaf8f0845d4b4c596c6fa087568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:08.911860 18333 raft_consensus.cc:1261] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138 [term 1 FOLLOWER]: Refusing update from remote peer deadeaaf8f0845d4b4c596c6fa087568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:08.912220 18400 consensus_queue.cc:1035] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83048b3bcd13469a9d870986d6e53138" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:08.912369 18403 consensus_queue.cc:1035] T 750b17cbf1674b968683ce44e3f00ad8 P deadeaaf8f0845d4b4c596c6fa087568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea974524fd44b0bb2deced006a400eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:08.914556 18406 mvcc.cc:205] Tried to move back new op lower bound from 7334179344021225472 to 7334179343976697856. Current Snapshot: MvccSnapshot[applied={T|T < 7334179344021225472}]
I20260928 04:52:08.914917 18407 mvcc.cc:205] Tried to move back new op lower bound from 7334179344021225472 to 7334179343976697856. Current Snapshot: MvccSnapshot[applied={T|T < 7334179344021225472}]
I20260928 04:52:08.916847 17929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:08.916934 17929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:08.917672 17929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 184caaca529d498484c7822a8a6c50c4: Sending DeleteTablet for 3 replicas of tablet 750b17cbf1674b968683ce44e3f00ad8
I20260928 04:52:08.917956 18051 tablet_service.cc:1514] Processing DeleteTablet for tablet 750b17cbf1674b968683ce44e3f00ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:08 UTC) from {username='slave'} at 127.0.0.1:35634
I20260928 04:52:08.918104 18182 tablet_service.cc:1514] Processing DeleteTablet for tablet 750b17cbf1674b968683ce44e3f00ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:08 UTC) from {username='slave'} at 127.0.0.1:39426
I20260928 04:52:08.918175 18313 tablet_service.cc:1514] Processing DeleteTablet for tablet 750b17cbf1674b968683ce44e3f00ad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:08 UTC) from {username='slave'} at 127.0.0.1:54114
I20260928 04:52:08.918278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 17989
I20260928 04:52:08.919210 18417 tablet_replica.cc:330] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: stopping tablet replica
I20260928 04:52:08.919317 18417 raft_consensus.cc:2229] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:52:08.919418 18417 raft_consensus.cc:2258] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:08.919807 18417 ts_tablet_manager.cc:1905] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:08.921257 18418 tablet_replica.cc:330] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: stopping tablet replica
I20260928 04:52:08.921358 18418 raft_consensus.cc:2229] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:52:08.921486 18418 raft_consensus.cc:2258] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:08.921679 18417 ts_tablet_manager.cc:1918] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:08.921751 18417 log.cc:1196] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/750b17cbf1674b968683ce44e3f00ad8
I20260928 04:52:08.921859 18418 ts_tablet_manager.cc:1905] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:08.922016 18417 ts_tablet_manager.cc:1939] T 750b17cbf1674b968683ce44e3f00ad8 P eea974524fd44b0bb2deced006a400eb: Deleting consensus metadata
I20260928 04:52:08.922713 17916 catalog_manager.cc:4841] TS eea974524fd44b0bb2deced006a400eb (127.0.98.2:46325): tablet 750b17cbf1674b968683ce44e3f00ad8 (table test-table [id=42c9ed8d31084488985425402b935cfb]) successfully deleted
I20260928 04:52:08.923170 18418 ts_tablet_manager.cc:1918] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:08.923225 18418 log.cc:1196] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/750b17cbf1674b968683ce44e3f00ad8
I20260928 04:52:08.923465 18418 ts_tablet_manager.cc:1939] T 750b17cbf1674b968683ce44e3f00ad8 P 83048b3bcd13469a9d870986d6e53138: Deleting consensus metadata
I20260928 04:52:08.923821 17913 catalog_manager.cc:4841] TS 83048b3bcd13469a9d870986d6e53138 (127.0.98.3:36787): tablet 750b17cbf1674b968683ce44e3f00ad8 (table test-table [id=42c9ed8d31084488985425402b935cfb]) successfully deleted
W20260928 04:52:08.924707 17913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35627 (error 108)
I20260928 04:52:08.924917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 18120
I20260928 04:52:08.925491 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:08.925062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35627 (local address 127.0.0.1:35656)
0928 04:52:08.925418 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:08.925466 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335}
W20260928 04:52:08.925630 17913 catalog_manager.cc:4584] TS deadeaaf8f0845d4b4c596c6fa087568 (127.0.98.1:35627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 750b17cbf1674b968683ce44e3f00ad8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35627: connect: Connection refused (error 111)
I20260928 04:52:08.930495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 18251
I20260928 04:52:08.934765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 17898
2026-09-28T04:52:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (683 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:08 dist-test-slave-jccs krb5kdc[18422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:08 dist-test-slave-jccs krb5kdc[18422](info): set up 2 sockets
Sep 28 04:52:08 dist-test-slave-jccs krb5kdc[18422](info): commencing operation
krb5kdc: starting...
W20260928 04:52:09.932859 392 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:10 dist-test-slave-jccs krb5kdc[18422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:10.044122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42295
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:10.123184 18435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:10.123425 18435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:10.123462 18435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:10.124569 18435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:10.124634 18435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:10.124660 18435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:10.124691 18435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:10.124714 18435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:10.126271 18435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:10.126528 18435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:10.126742 18435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:10.131210 18441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:10.131242 18443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:10.131218 18440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:10.131455 18435 server_base.cc:1034] running on GCE node
I20260928 04:52:10.131729 18435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:10.132040 18435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:10.133229 18435 hybrid_clock.cc:648] HybridClock initialized: now 1790571130133216 us; error 43 us; skew 500 ppm
Sep 28 04:52:10 dist-test-slave-jccs krb5kdc[18422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:10.135654 18435 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:10.136075 18435 webserver.cc:460] Webserver started at http://127.0.98.62:34687/ using document root <none> and password file <none>
I20260928 04:52:10.136234 18435 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:10.136271 18435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:10.136348 18435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:10.137284 18435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "39f61c5944fb41c69ad6008db23d1033"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "9a20eea3340fee790220bf6f943ac907"
server_key_iv: "5352b67797c8e8efe505ea10ccf3d8f3"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.137571 18435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "39f61c5944fb41c69ad6008db23d1033"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "9a20eea3340fee790220bf6f943ac907"
server_key_iv: "5352b67797c8e8efe505ea10ccf3d8f3"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.138756 18435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:52:10.139358 18450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:10.139540 18435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:52:10.139621 18435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "39f61c5944fb41c69ad6008db23d1033"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "9a20eea3340fee790220bf6f943ac907"
server_key_iv: "5352b67797c8e8efe505ea10ccf3d8f3"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.139670 18435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:10.177634 18435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:10.179917 18435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:10.180027 18435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:10.183914 18435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42295
I20260928 04:52:10.184001 18502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42295 every 8 connection(s)
I20260928 04:52:10.184340 18435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:10.185298 18503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:10.187582 18503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033: Bootstrap starting.
I20260928 04:52:10.188175 18503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:10.188488 18503 log.cc:824] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:10.188859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 18435
I20260928 04:52:10.189029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:10.189278 392 external_mini_cluster.cc:1428] Setting key b00ac4891e25c453280a9545be10e32d
I20260928 04:52:10.189303 18503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033: No bootstrap required, opened a new log
I20260928 04:52:10.191184 18503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f61c5944fb41c69ad6008db23d1033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42295 } }
I20260928 04:52:10.191362 18503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:10.191404 18503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f61c5944fb41c69ad6008db23d1033, State: Initialized, Role: FOLLOWER
I20260928 04:52:10.191560 18503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f61c5944fb41c69ad6008db23d1033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42295 } }
I20260928 04:52:10.191663 18503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:10.191697 18503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:10.191762 18503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:10.192715 18503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f61c5944fb41c69ad6008db23d1033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42295 } }
I20260928 04:52:10.192806 18503 leader_election.cc:302] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39f61c5944fb41c69ad6008db23d1033; no voters:
I20260928 04:52:10.193011 18503 leader_election.cc:288] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:10.193030 18508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:10.193141 18508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [term 1 LEADER]: Becoming Leader. State: Replica: 39f61c5944fb41c69ad6008db23d1033, State: Running, Role: LEADER
I20260928 04:52:10.193305 18503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:10.193312 18508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f61c5944fb41c69ad6008db23d1033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42295 } }
I20260928 04:52:10.193758 18509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39f61c5944fb41c69ad6008db23d1033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f61c5944fb41c69ad6008db23d1033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42295 } } }
I20260928 04:52:10.193763 18510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39f61c5944fb41c69ad6008db23d1033. Latest consensus state: current_term: 1 leader_uuid: "39f61c5944fb41c69ad6008db23d1033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39f61c5944fb41c69ad6008db23d1033" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42295 } } }
I20260928 04:52:10.193902 18510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:10.193858 18509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:10.194185 18515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:10.194828 18515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:10.196305 18515 catalog_manager.cc:1347] Generated new cluster ID: 011358fd7bf7401782f492b2b6fe4dce
I20260928 04:52:10.196360 18515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:52:10 dist-test-slave-jccs krb5kdc[18422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:10.201406 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.189791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.0.98.62:42295)
0928 04:52:10.189956 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.189971 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.189998 (+ 27us) server_negotiation.cc:407] Connection header received
0928 04:52:10.191167 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.191173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.191218 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.191329 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:10.192195 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.193319 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.194414 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.195170 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.195270 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.197769 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:10.197773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:10.197773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:10.197784 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:10.199486 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:10.199860 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:10.199863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:10.199863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:10.199973 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:10.200184 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:10.200185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:10.200185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:10.200404 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:10.200990 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.201044 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.201143 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260928 04:52:10.202313 18515 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:10.202828 18515 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:10.212497 18515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033: Generated new TSK 0
I20260928 04:52:10.212679 18515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:52:10.240435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42295
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:10.326140 18529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:10.326370 18529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:10.326406 18529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:10.327487 18529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:10.327554 18529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:10.327598 18529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:52:10.329130 18529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:10.329396 18529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:10.329615 18529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:10.335048 18535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:10.335055 18534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:10.335263 18537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:10.335505 18529 server_base.cc:1034] running on GCE node
I20260928 04:52:10.335709 18529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:10.335965 18529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:10.337159 18529 hybrid_clock.cc:648] HybridClock initialized: now 1790571130337141 us; error 27 us; skew 500 ppm
Sep 28 04:52:10 dist-test-slave-jccs krb5kdc[18422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:10.339591 18529 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:52:10.340027 18529 webserver.cc:460] Webserver started at http://127.0.98.1:45773/ using document root <none> and password file <none>
I20260928 04:52:10.340248 18529 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:10.340293 18529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:10.340444 18529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:10.341516 18529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "c6204d509eb84603ac96e56578ae7c3c"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "99be51107a65f916feaf19ba0a17fbe3"
server_key_iv: "5fffa1dd2eb6c300952e6f3b61bbb2eb"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.341893 18529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c6204d509eb84603ac96e56578ae7c3c"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "99be51107a65f916feaf19ba0a17fbe3"
server_key_iv: "5fffa1dd2eb6c300952e6f3b61bbb2eb"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.343318 18529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:52:10.344029 18544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:10.344192 18529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:10.344285 18529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "c6204d509eb84603ac96e56578ae7c3c"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "99be51107a65f916feaf19ba0a17fbe3"
server_key_iv: "5fffa1dd2eb6c300952e6f3b61bbb2eb"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.344363 18529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:10.387646 18529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:10.389631 18529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:10.389792 18529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:10.390025 18529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:10.390367 18529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:10.390424 18529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:10.390466 18529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:10.390496 18529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:10.396432 18529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42729
I20260928 04:52:10.396530 18657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42729 every 8 connection(s)
I20260928 04:52:10.396874 18529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 04:52:10 dist-test-slave-jccs krb5kdc[18422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:10.405791 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.397621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40307 (local address 127.0.98.62:42295)
0928 04:52:10.397684 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.397686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.397898 (+ 212us) server_negotiation.cc:407] Connection header received
0928 04:52:10.398868 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.398869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.398881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.398935 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:10.399535 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.400024 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.400987 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.401270 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.401346 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.403458 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:10.403460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:10.403460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:10.403469 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:10.404609 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:10.405013 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:10.405013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:10.405014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:10.405061 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:10.405311 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:10.405311 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:10.405312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:10.405420 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:10.405464 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.405668 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.405733 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:52:10.405946 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.397605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42295 (local address 127.0.98.1:40307)
0928 04:52:10.397764 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:10.397798 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:10.398759 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:10.399002 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:10.399005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:10.399259 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:10.399439 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.399443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.400116 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.400117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.400898 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.400899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.401356 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.401356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.401491 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.402008 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:10.402016 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:10.403370 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:10.404683 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:10.404690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:10.404692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:10.404934 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:10.405116 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:10.405116 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:10.405117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:10.405261 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:10.405469 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:10.405473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:10.405570 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 04:52:10.405642 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 04:52:10.405741 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 04:52:10.406333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 18529
I20260928 04:52:10.406451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:52:10.406461 18658 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42295
I20260928 04:52:10.406549 18658 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:10.406665 392 external_mini_cluster.cc:1428] Setting key b3947b3a504fd33cd4853390203dd1c9
I20260928 04:52:10.406770 18658 heartbeater.cc:511] Master 127.0.98.62:42295 requested a full tablet report, sending...
I20260928 04:52:10.407217 18466 ts_manager.cc:194] Registered new tserver with Master: c6204d509eb84603ac96e56578ae7c3c (127.0.98.1:42729)
I20260928 04:52:10.407853 18466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40307
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:52:10.431465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42295
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:10.509146 18663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:10.509349 18663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:10.509370 18663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:10.510444 18663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:10.510479 18663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:10.510504 18663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:52:10.511885 18663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:10.512074 18663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:10.512229 18663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:10.516888 18671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:10.516917 18669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:10.516888 18668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:10.517162 18663 server_base.cc:1034] running on GCE node
I20260928 04:52:10.517545 18663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:10.517761 18663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:10.518977 18663 hybrid_clock.cc:648] HybridClock initialized: now 1790571130518970 us; error 32 us; skew 500 ppm
Sep 28 04:52:10 dist-test-slave-jccs krb5kdc[18422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:10.521399 18663 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:52:10.521823 18663 webserver.cc:460] Webserver started at http://127.0.98.2:37771/ using document root <none> and password file <none>
I20260928 04:52:10.522043 18663 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:10.522105 18663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:10.522267 18663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:10.523316 18663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "c970b23d9c714974bcf9a4563cce257e"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "2b1aab4e1b7d4db9eb77a08e995fac3c"
server_key_iv: "bdcc49d7ef0504b1d793b23d50a26010"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.523690 18663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c970b23d9c714974bcf9a4563cce257e"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "2b1aab4e1b7d4db9eb77a08e995fac3c"
server_key_iv: "bdcc49d7ef0504b1d793b23d50a26010"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.524932 18663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:52:10.525691 18678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:10.525962 18663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 04:52:10.526060 18663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "c970b23d9c714974bcf9a4563cce257e"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "2b1aab4e1b7d4db9eb77a08e995fac3c"
server_key_iv: "bdcc49d7ef0504b1d793b23d50a26010"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.526140 18663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:10.544543 18663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:10.546839 18663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:10.546988 18663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:10.547204 18663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:10.547487 18663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:10.547538 18663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:10.547578 18663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:10.547605 18663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:10.553476 18663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44515
I20260928 04:52:10.553537 18791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44515 every 8 connection(s)
I20260928 04:52:10.553825 18663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:52:10.556090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 18663
I20260928 04:52:10.556205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:52:10.556427 392 external_mini_cluster.cc:1428] Setting key 0130816431576793c15d8aa4b3758616
Sep 28 04:52:10 dist-test-slave-jccs krb5kdc[18422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:10.564087 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.554366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42295 (local address 127.0.98.2:44311)
0928 04:52:10.554545 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:10.554592 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:10.555529 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:10.555806 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:10.555810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:10.556227 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:10.556504 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.556522 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.557303 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.557304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.558582 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.558585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.559062 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.559063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.559159 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.559613 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:10.559621 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:10.560895 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:10.562839 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:10.562845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:10.562847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:10.563058 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:10.563270 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:10.563270 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:10.563270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:10.563359 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:10.563653 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:10.563656 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:10.563765 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 04:52:10.563839 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 04:52:10.563944 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260928 04:52:10.564088 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.554430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44311 (local address 127.0.98.62:42295)
0928 04:52:10.554491 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.554493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.554686 (+ 193us) server_negotiation.cc:407] Connection header received
0928 04:52:10.555663 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.555665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.555677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.555733 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:10.556637 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.557228 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.558668 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.558982 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.559055 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.560988 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:10.560990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:10.560990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:10.560998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:10.562749 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:10.563131 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:10.563133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:10.563133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:10.563199 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:10.563425 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:10.563426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:10.563426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:10.563585 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:10.563665 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.563888 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.563973 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:52:10.564518 18792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42295
I20260928 04:52:10.564627 18792 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:10.564790 18792 heartbeater.cc:511] Master 127.0.98.62:42295 requested a full tablet report, sending...
I20260928 04:52:10.565258 18466 ts_manager.cc:194] Registered new tserver with Master: c970b23d9c714974bcf9a4563cce257e (127.0.98.2:44515)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:52:10.565765 18466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:52:10.582103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42295
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:10.659865 18797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:10.660158 18797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:10.660223 18797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:10.661441 18797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:10.661486 18797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:10.661512 18797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:52:10.663215 18797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:10.663394 18797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:10.663554 18797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:10.668350 18803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:10.668357 18805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:10.668386 18797 server_base.cc:1034] running on GCE node
W20260928 04:52:10.668362 18802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:10.668730 18797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:10.668967 18797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:10.670100 18797 hybrid_clock.cc:648] HybridClock initialized: now 1790571130670089 us; error 28 us; skew 500 ppm
Sep 28 04:52:10 dist-test-slave-jccs krb5kdc[18422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:10.672554 18797 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:52:10.672981 18797 webserver.cc:460] Webserver started at http://127.0.98.3:40553/ using document root <none> and password file <none>
I20260928 04:52:10.673272 18797 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:10.673338 18797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:10.673467 18797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:10.674453 18797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "bc99bf626ae1479db4a483f91c3521cf"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "069a27c4b33ef5f9e7fdadd64755f8dc"
server_key_iv: "976058a1d7bb1dd4fdbf65079aacc555"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.674799 18797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bc99bf626ae1479db4a483f91c3521cf"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "069a27c4b33ef5f9e7fdadd64755f8dc"
server_key_iv: "976058a1d7bb1dd4fdbf65079aacc555"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.676059 18797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:10.676772 18812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:10.676935 18797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:10.677027 18797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "bc99bf626ae1479db4a483f91c3521cf"
format_stamp: "Formatted at 2026-09-28 04:52:10 on dist-test-slave-jccs"
server_key: "069a27c4b33ef5f9e7fdadd64755f8dc"
server_key_iv: "976058a1d7bb1dd4fdbf65079aacc555"
server_key_version: "encryptionkey@0"
I20260928 04:52:10.677124 18797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:10.703205 18797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:10.705616 18797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:10.705773 18797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:10.706007 18797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:10.706321 18797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:10.706382 18797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:10.706418 18797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:10.706446 18797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:10.712055 18797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34291
I20260928 04:52:10.712102 18925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34291 every 8 connection(s)
I20260928 04:52:10.712419 18797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:52:10.716679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 18797
I20260928 04:52:10.716787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:52:10.716996 392 external_mini_cluster.cc:1428] Setting key 2cb00dee9914dfd3cdd787fc6d7fd2f6
Sep 28 04:52:10 dist-test-slave-jccs krb5kdc[18422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:10.721095 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.713130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52045 (local address 127.0.98.62:42295)
0928 04:52:10.713209 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.713211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.713403 (+ 192us) server_negotiation.cc:407] Connection header received
0928 04:52:10.714389 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.714390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.714404 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.714453 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:10.715064 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.715563 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.716545 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.716757 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.716838 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.718936 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:10.718938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:10.718938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:10.718944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:10.720019 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:10.720373 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:10.720374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:10.720374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:10.720421 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:10.720618 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:10.720618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:10.720619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:10.720712 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:10.720759 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.720977 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.721036 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:52:10.721324 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.713108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42295 (local address 127.0.98.3:52045)
0928 04:52:10.713288 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:10.713323 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:10.714256 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:10.714518 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:10.714522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:10.714787 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:10.714969 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.714973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.715657 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.715658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.716454 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.716456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.716863 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.716864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.717024 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.717595 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:10.717603 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:10.718853 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:10.720090 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:10.720095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:10.720097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:10.720306 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:10.720476 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:10.720476 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:10.720476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:10.720566 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:10.720764 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:10.720768 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:10.720872 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 04:52:10.720950 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:52:10.721058 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260928 04:52:10.721823 18926 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42295
I20260928 04:52:10.721925 18926 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:10.722102 18926 heartbeater.cc:511] Master 127.0.98.62:42295 requested a full tablet report, sending...
I20260928 04:52:10.722492 18466 ts_manager.cc:194] Registered new tserver with Master: bc99bf626ae1479db4a483f91c3521cf (127.0.98.3:34291)
I20260928 04:52:10.722855 18466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52045
I20260928 04:52:10.728492 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:52:10.735234 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.729850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.0.98.62:42295)
0928 04:52:10.729900 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.729902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.730014 (+ 112us) server_negotiation.cc:407] Connection header received
0928 04:52:10.730052 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.730053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.730060 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.730104 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:10.730585 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.731022 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.731694 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.731865 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.731916 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.733115 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:10.733116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:10.733116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:10.733121 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:10.734248 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:10.734582 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:10.734583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:10.734583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:10.734629 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:10.734833 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:10.734833 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:10.734833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:10.734955 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:10.735030 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.735136 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.735179 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:52:10.736603 18466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:52:10.738161 18466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:52:10.744213 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.740338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44515 (local address 127.0.0.1:33186)
0928 04:52:10.740616 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:10.740626 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:10.740667 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:10.740898 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:10.740899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:10.740902 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:10.740946 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.740966 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.742308 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.742309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.743209 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.743212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.743926 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.743927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.744002 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.744014 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 04:52:10.744092 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 04:52:10.744144 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":58,"threads_started":1}
I20260928 04:52:10.744225 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.740461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.0.98.2:44515)
0928 04:52:10.740653 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.740656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.740666 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:52:10.740733 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.740735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.740749 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.740818 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:10.741092 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.742211 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.743303 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.743837 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.743925 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.743984 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.744113 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.744153 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260928 04:52:10.744405 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.740120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34291 (local address 127.0.0.1:51188)
0928 04:52:10.740243 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:10.740280 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:10.740358 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:10.740772 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:10.740774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:10.740776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:10.740848 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.740851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.741916 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.741917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.743227 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.743230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.744164 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.744164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.744238 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.744240 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:52:10.744270 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 04:52:10.744317 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":55,"thread_start_us":77,"threads_started":1}
I20260928 04:52:10.744704 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.740153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42729 (local address 127.0.0.1:59692)
0928 04:52:10.740456 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:10.740466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:10.740548 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:10.740833 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:10.740834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:10.740836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:10.740864 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.740865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.742212 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.742213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.743314 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.743316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.744162 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.744164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.744235 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.744239 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:52:10.744587 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 04:52:10.744626 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260928 04:52:10.744807 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.740460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.0.98.3:34291)
0928 04:52:10.740614 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.740619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.740630 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:52:10.740670 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.740672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.740687 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.740771 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:10.740906 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.741821 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.743321 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.744056 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.744567 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.744641 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.744666 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.744716 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260928 04:52:10.744938 18726 tablet_service.cc:1467] Processing CreateTablet for tablet b18ad4ac0588408ab486654526c277c9 (DEFAULT_TABLE table=test-table [id=c2ee3fa927c842a29b93645d104c0762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:10.745107 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.740281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.0.98.1:42729)
0928 04:52:10.740670 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.740674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.740683 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:52:10.740725 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.740727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.740742 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.740931 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:10.741028 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.742104 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.743395 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.744059 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.744892 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.744961 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.744983 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.745024 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":89,"threads_started":1}
I20260928 04:52:10.745324 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18ad4ac0588408ab486654526c277c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:10.748021 18943 tablet_bootstrap.cc:492] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e: Bootstrap starting.
I20260928 04:52:10.748373 18860 tablet_service.cc:1467] Processing CreateTablet for tablet b18ad4ac0588408ab486654526c277c9 (DEFAULT_TABLE table=test-table [id=c2ee3fa927c842a29b93645d104c0762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:10.748710 18943 tablet_bootstrap.cc:654] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:10.748718 18860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18ad4ac0588408ab486654526c277c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:10.749117 18943 log.cc:824] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:10.749835 18943 tablet_bootstrap.cc:492] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e: No bootstrap required, opened a new log
I20260928 04:52:10.749902 18943 ts_tablet_manager.cc:1397] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:52:10.751152 18943 raft_consensus.cc:348] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 } } peers { permanent_uuid: "c970b23d9c714974bcf9a4563cce257e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44515 } } peers { permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 } }
I20260928 04:52:10.751319 18943 raft_consensus.cc:374] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:10.751368 18943 raft_consensus.cc:729] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c970b23d9c714974bcf9a4563cce257e, State: Initialized, Role: FOLLOWER
I20260928 04:52:10.751261 18945 tablet_bootstrap.cc:492] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: Bootstrap starting.
I20260928 04:52:10.751483 18943 consensus_queue.cc:260] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 } } peers { permanent_uuid: "c970b23d9c714974bcf9a4563cce257e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44515 } } peers { permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 } }
I20260928 04:52:10.751757 18943 ts_tablet_manager.cc:1428] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 04:52:10.751873 18792 heartbeater.cc:503] Master 127.0.98.62:42295 was elected leader, sending a full tablet report...
I20260928 04:52:10.751853 18592 tablet_service.cc:1467] Processing CreateTablet for tablet b18ad4ac0588408ab486654526c277c9 (DEFAULT_TABLE table=test-table [id=c2ee3fa927c842a29b93645d104c0762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:10.752137 18945 tablet_bootstrap.cc:654] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:10.752173 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18ad4ac0588408ab486654526c277c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:10.752691 18945 log.cc:824] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:10.753518 18945 tablet_bootstrap.cc:492] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: No bootstrap required, opened a new log
I20260928 04:52:10.753644 18945 ts_tablet_manager.cc:1397] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:52:10.754907 18948 tablet_bootstrap.cc:492] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c: Bootstrap starting.
I20260928 04:52:10.755016 18945 raft_consensus.cc:348] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 } } peers { permanent_uuid: "c970b23d9c714974bcf9a4563cce257e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44515 } } peers { permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 } }
I20260928 04:52:10.755188 18945 raft_consensus.cc:374] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:10.755240 18945 raft_consensus.cc:729] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc99bf626ae1479db4a483f91c3521cf, State: Initialized, Role: FOLLOWER
I20260928 04:52:10.755349 18945 consensus_queue.cc:260] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 } } peers { permanent_uuid: "c970b23d9c714974bcf9a4563cce257e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44515 } } peers { permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 } }
I20260928 04:52:10.755544 18948 tablet_bootstrap.cc:654] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:10.755607 18945 ts_tablet_manager.cc:1428] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:52:10.755628 18926 heartbeater.cc:503] Master 127.0.98.62:42295 was elected leader, sending a full tablet report...
I20260928 04:52:10.756107 18948 log.cc:824] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:10.756783 18948 tablet_bootstrap.cc:492] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c: No bootstrap required, opened a new log
I20260928 04:52:10.756865 18948 ts_tablet_manager.cc:1397] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:52:10.758337 18948 raft_consensus.cc:348] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 } } peers { permanent_uuid: "c970b23d9c714974bcf9a4563cce257e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44515 } } peers { permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 } }
I20260928 04:52:10.758487 18948 raft_consensus.cc:374] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:10.758533 18948 raft_consensus.cc:729] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6204d509eb84603ac96e56578ae7c3c, State: Initialized, Role: FOLLOWER
I20260928 04:52:10.758638 18948 consensus_queue.cc:260] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 } } peers { permanent_uuid: "c970b23d9c714974bcf9a4563cce257e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44515 } } peers { permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 } }
I20260928 04:52:10.758905 18658 heartbeater.cc:503] Master 127.0.98.62:42295 was elected leader, sending a full tablet report...
I20260928 04:52:10.758903 18948 ts_tablet_manager.cc:1428] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 04:52:10.804315 18793 tablet.cc:2369] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:52:10.897588 18659 tablet.cc:2369] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:52:10.929790 18946 raft_consensus.cc:482] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:52:10.929977 18946 raft_consensus.cc:504] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 } } peers { permanent_uuid: "c970b23d9c714974bcf9a4563cce257e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44515 } } peers { permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 } }
I20260928 04:52:10.930339 18946 leader_election.cc:288] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc99bf626ae1479db4a483f91c3521cf (127.0.98.3:34291), c6204d509eb84603ac96e56578ae7c3c (127.0.98.1:42729)
I20260928 04:52:10.934327 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.930610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42729 (local address 127.0.98.2:44865)
0928 04:52:10.930685 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:10.930699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:10.930788 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:10.931092 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:10.931092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:10.931094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:10.931162 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.931164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.932033 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.932034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.933467 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.933470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.934065 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.934066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.934144 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.934147 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:10.934216 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 04:52:10.934254 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 04:52:10.934334 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.930902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44865 (local address 127.0.98.1:42729)
0928 04:52:10.930976 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.930978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.930994 (+ 16us) server_negotiation.cc:407] Connection header received
0928 04:52:10.931024 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.931025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.931037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.931089 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:10.931232 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.931959 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.933571 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.933969 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.934066 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.934077 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.934231 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.934266 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 04:52:10.934711 18612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18ad4ac0588408ab486654526c277c9" candidate_uuid: "c970b23d9c714974bcf9a4563cce257e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6204d509eb84603ac96e56578ae7c3c" is_pre_election: true
I20260928 04:52:10.934913 18612 raft_consensus.cc:2454] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c970b23d9c714974bcf9a4563cce257e in term 0.
I20260928 04:52:10.935161 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.930822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50899 (local address 127.0.98.3:34291)
0928 04:52:10.930869 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.930871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.930986 (+ 115us) server_negotiation.cc:407] Connection header received
0928 04:52:10.931137 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.931138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.931149 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.931222 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:10.931552 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.932174 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.933703 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.934291 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.934366 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.934377 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.935051 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.935100 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:52:10.935158 18681 leader_election.cc:302] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6204d509eb84603ac96e56578ae7c3c, c970b23d9c714974bcf9a4563cce257e; no voters:
I20260928 04:52:10.935191 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.930745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34291 (local address 127.0.98.2:50899)
0928 04:52:10.930920 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:10.930929 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:10.930996 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:10.931347 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:10.931348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:10.931351 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:10.931438 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.931440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.932277 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.932278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.933613 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:10.933615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.934730 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.934731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:10.934812 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.934816 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:52:10.935040 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 04:52:10.935085 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260928 04:52:10.935262 18946 raft_consensus.cc:2781] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:52:10.935566 18946 raft_consensus.cc:482] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:52:10.935606 18946 raft_consensus.cc:3037] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:10.935817 18880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18ad4ac0588408ab486654526c277c9" candidate_uuid: "c970b23d9c714974bcf9a4563cce257e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc99bf626ae1479db4a483f91c3521cf" is_pre_election: true
I20260928 04:52:10.935973 18880 raft_consensus.cc:2454] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c970b23d9c714974bcf9a4563cce257e in term 0.
I20260928 04:52:10.936546 18946 raft_consensus.cc:504] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 } } peers { permanent_uuid: "c970b23d9c714974bcf9a4563cce257e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44515 } } peers { permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 } }
I20260928 04:52:10.936698 18946 leader_election.cc:288] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [CANDIDATE]: Term 1 election: Requested vote from peers bc99bf626ae1479db4a483f91c3521cf (127.0.98.3:34291), c6204d509eb84603ac96e56578ae7c3c (127.0.98.1:42729)
I20260928 04:52:10.937063 18612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18ad4ac0588408ab486654526c277c9" candidate_uuid: "c970b23d9c714974bcf9a4563cce257e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6204d509eb84603ac96e56578ae7c3c"
I20260928 04:52:10.937072 18880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18ad4ac0588408ab486654526c277c9" candidate_uuid: "c970b23d9c714974bcf9a4563cce257e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc99bf626ae1479db4a483f91c3521cf"
I20260928 04:52:10.937167 18612 raft_consensus.cc:3037] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:10.937211 18880 raft_consensus.cc:3037] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:10.937772 18612 raft_consensus.cc:2454] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c970b23d9c714974bcf9a4563cce257e in term 1.
I20260928 04:52:10.937906 18681 leader_election.cc:302] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6204d509eb84603ac96e56578ae7c3c, c970b23d9c714974bcf9a4563cce257e; no voters:
I20260928 04:52:10.937950 18880 raft_consensus.cc:2454] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c970b23d9c714974bcf9a4563cce257e in term 1.
I20260928 04:52:10.938002 18946 raft_consensus.cc:2781] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:10.938133 18946 raft_consensus.cc:686] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 1 LEADER]: Becoming Leader. State: Replica: c970b23d9c714974bcf9a4563cce257e, State: Running, Role: LEADER
I20260928 04:52:10.938215 18946 consensus_queue.cc:237] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 } } peers { permanent_uuid: "c970b23d9c714974bcf9a4563cce257e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44515 } } peers { permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 } }
I20260928 04:52:10.938963 18466 catalog_manager.cc:5495] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e reported cstate change: term changed from 0 to 1, leader changed from <none> to c970b23d9c714974bcf9a4563cce257e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c970b23d9c714974bcf9a4563cce257e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c970b23d9c714974bcf9a4563cce257e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 } health_report { overall_health: UNKNOWN } } }
W20260928 04:52:10.963028 18927 tablet.cc:2369] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:52:10.970343 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.968154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.0.98.2:44515)
0928 04:52:10.968226 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:10.968228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:10.968240 (+ 12us) server_negotiation.cc:407] Connection header received
0928 04:52:10.968275 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:10.968276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:10.968295 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:10.968355 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:10.968467 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.968945 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.969697 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:10.969919 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:10.969977 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:10.970090 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:10.970171 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:10.970238 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:10.970279 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 04:52:10.970289 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:52:10.972206 18880 raft_consensus.cc:1261] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf [term 1 FOLLOWER]: Refusing update from remote peer c970b23d9c714974bcf9a4563cce257e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:10.972206 18612 raft_consensus.cc:1261] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c [term 1 FOLLOWER]: Refusing update from remote peer c970b23d9c714974bcf9a4563cce257e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:10.972555 18946 consensus_queue.cc:1035] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc99bf626ae1479db4a483f91c3521cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:10.972745 18953 consensus_queue.cc:1035] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6204d509eb84603ac96e56578ae7c3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:10.974749 18956 mvcc.cc:205] Tried to move back new op lower bound from 7334179352459927552 to 7334179352324349952. Current Snapshot: MvccSnapshot[applied={T|T < 7334179352459927552}]
I20260928 04:52:10.975630 18958 mvcc.cc:205] Tried to move back new op lower bound from 7334179352459927552 to 7334179352324349952. Current Snapshot: MvccSnapshot[applied={T|T < 7334179352459927552}]
I20260928 04:52:10.977474 18467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:10.977607 18467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:10.978353 18467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39f61c5944fb41c69ad6008db23d1033: Sending DeleteTablet for 3 replicas of tablet b18ad4ac0588408ab486654526c277c9
I20260928 04:52:10.978663 18860 tablet_service.cc:1514] Processing DeleteTablet for tablet b18ad4ac0588408ab486654526c277c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51188
I20260928 04:52:10.978684 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet b18ad4ac0588408ab486654526c277c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59692
I20260928 04:52:10.978883 18966 tablet_replica.cc:330] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: stopping tablet replica
I20260928 04:52:10.978943 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet b18ad4ac0588408ab486654526c277c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33186
I20260928 04:52:10.978998 18966 raft_consensus.cc:2229] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:52:10.979097 18967 tablet_replica.cc:330] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c: stopping tablet replica
I20260928 04:52:10.979132 18966 raft_consensus.cc:2258] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:10.979178 18967 raft_consensus.cc:2229] T b18ad4ac0588408ab486654526c277c9 P c6204d509eb84603ac96e56578ae7c3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:52:10.979334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 18529
I20260928 04:52:10.981093 18966 ts_tablet_manager.cc:1905] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:10.983115 18966 ts_tablet_manager.cc:1918] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:10.983192 18966 log.cc:1196] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/b18ad4ac0588408ab486654526c277c9
I20260928 04:52:10.983490 18966 ts_tablet_manager.cc:1939] T b18ad4ac0588408ab486654526c277c9 P bc99bf626ae1479db4a483f91c3521cf: Deleting consensus metadata
I20260928 04:52:10.983942 18454 catalog_manager.cc:4841] TS bc99bf626ae1479db4a483f91c3521cf (127.0.98.3:34291): tablet b18ad4ac0588408ab486654526c277c9 (table test-table [id=c2ee3fa927c842a29b93645d104c0762]) successfully deleted
W20260928 04:52:10.985967 18453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42729 (error 108)
I20260928 04:52:10.986050 18968 tablet_replica.cc:330] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e: stopping tablet replica
I20260928 04:52:10.986137 18968 raft_consensus.cc:2229] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:52:10.986256 18968 raft_consensus.cc:2258] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:10.986491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 18663
I20260928 04:52:10.986773 18968 ts_tablet_manager.cc:1905] T b18ad4ac0588408ab486654526c277c9 P c970b23d9c714974bcf9a4563cce257e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:10.987823 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.987674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42729 (local address 127.0.0.1:59702)
0928 04:52:10.987743 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:10.987784 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 04:52:10.987942 18453 catalog_manager.cc:4584] TS c6204d509eb84603ac96e56578ae7c3c (127.0.98.1:42729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b18ad4ac0588408ab486654526c277c9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42729: connect: Connection refused (error 111)
I20260928 04:52:10.991470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 18797
I20260928 04:52:10.991575 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:10.991442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44515 (local address 127.0.0.1:33208)
0928 04:52:10.991497 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:10.991542 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 04:52:10.995930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 18435
2026-09-28T04:52:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2062 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T04:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:11Z Disabled control of system clock
I20260928 04:52:11.022639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44033
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:52:11.102469 18973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:11.102670 18973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:11.102690 18973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:11.103744 18973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:11.103778 18973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:11.103790 18973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:11.103802 18973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:11.103813 18973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:11.105010 18973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:11.105239 18973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:11.105484 18973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:11.109899 18979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:11.109938 18978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:11.110039 18973 server_base.cc:1034] running on GCE node
W20260928 04:52:11.110162 18981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:11.110417 18973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:11.110694 18973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:11.111832 18973 hybrid_clock.cc:648] HybridClock initialized: now 1790571131111823 us; error 27 us; skew 500 ppm
I20260928 04:52:11.113075 18973 webserver.cc:460] Webserver started at http://127.0.98.62:34865/ using document root <none> and password file <none>
I20260928 04:52:11.113355 18973 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:11.113399 18973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:11.113554 18973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:11.114609 18973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "209a67cf9b8a5ec130a233e37211b6d0"
server_key_iv: "89fbfb6642d97f5ad7a6977611ea462f"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.114969 18973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "209a67cf9b8a5ec130a233e37211b6d0"
server_key_iv: "89fbfb6642d97f5ad7a6977611ea462f"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.116241 18973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:11.116909 18987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:11.117092 18973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:11.117203 18973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "209a67cf9b8a5ec130a233e37211b6d0"
server_key_iv: "89fbfb6642d97f5ad7a6977611ea462f"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.117278 18973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:11.131809 18973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:11.132084 18973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:11.132248 18973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:11.136153 18973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44033
I20260928 04:52:11.136205 19039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44033 every 8 connection(s)
I20260928 04:52:11.136503 18973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:11.137377 19040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:11.139438 19040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86: Bootstrap starting.
I20260928 04:52:11.139963 19040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:11.140259 19040 log.cc:824] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:11.140914 19040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86: No bootstrap required, opened a new log
I20260928 04:52:11.142462 19040 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260928 04:52:11.142593 19040 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:11.142616 19040 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8bee9ebfc043ddb7d8f1e3d4472f86, State: Initialized, Role: FOLLOWER
I20260928 04:52:11.142777 19040 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260928 04:52:11.142858 19040 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:11.142917 19040 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:11.142966 19040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:11.143487 19040 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260928 04:52:11.143599 19040 leader_election.cc:302] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea8bee9ebfc043ddb7d8f1e3d4472f86; no voters:
I20260928 04:52:11.143780 19040 leader_election.cc:288] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:11.143836 19043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:11.144007 19043 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [term 1 LEADER]: Becoming Leader. State: Replica: ea8bee9ebfc043ddb7d8f1e3d4472f86, State: Running, Role: LEADER
I20260928 04:52:11.144057 19040 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:11.144110 19043 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } }
I20260928 04:52:11.144486 19044 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } } }
I20260928 04:52:11.144579 19044 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:11.144565 19045 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea8bee9ebfc043ddb7d8f1e3d4472f86. Latest consensus state: current_term: 1 leader_uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8bee9ebfc043ddb7d8f1e3d4472f86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44033 } } }
I20260928 04:52:11.144635 19045 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:11.144809 19052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:11.145346 19052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:11.146551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 18973
I20260928 04:52:11.146670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:11.146889 392 external_mini_cluster.cc:1428] Setting key 0ab04de5b1a074eb1a8819c9583b9cfa
I20260928 04:52:11.146994 19052 catalog_manager.cc:1347] Generated new cluster ID: 3ab4a506be15479191ddc329330c613c
I20260928 04:52:11.147038 19052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:11.154534 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.147361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.0.98.62:44033)
0928 04:52:11.149331 (+ 1970us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.149341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.149374 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:52:11.149484 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.149491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.149557 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.149699 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:11.149943 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.151331 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.152387 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.153045 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.153197 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.153858 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:11.153862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:11.153862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:11.154072 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:11.154113 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:11.154127 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:11.154190 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.154266 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.154338 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1902,"thread_start_us":65,"threads_started":1}
I20260928 04:52:11.156078 19052 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:11.156708 19052 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:11.174247 19052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86: Generated new TSK 0
I20260928 04:52:11.174453 19052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:11.179207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--builtin_ntp_servers=127.0.98.20:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:52:11.269120 19064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:11.269330 19064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:11.269351 19064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:11.270494 19064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:11.270532 19064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:11.270557 19064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:52:11.271952 19064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:11.272189 19064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:11.272478 19064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:11.277124 19070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:11.277190 19069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:11.277357 19064 server_base.cc:1034] running on GCE node
W20260928 04:52:11.277369 19072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:11.277629 19064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:11.277876 19064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:11.279050 19064 hybrid_clock.cc:648] HybridClock initialized: now 1790571131279032 us; error 32 us; skew 500 ppm
I20260928 04:52:11.280323 19064 webserver.cc:460] Webserver started at http://127.0.98.1:42067/ using document root <none> and password file <none>
I20260928 04:52:11.280553 19064 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:11.280602 19064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:11.280707 19064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:11.281967 19064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "88b8d499a36c48389a7827642edd8e63"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "c7ced29cf851d6691ab400f3fc504d10"
server_key_iv: "5ba8a9537ffcc98ecc0c78ace90eea11"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.282377 19064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "88b8d499a36c48389a7827642edd8e63"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "c7ced29cf851d6691ab400f3fc504d10"
server_key_iv: "5ba8a9537ffcc98ecc0c78ace90eea11"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.283939 19064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:52:11.284765 19078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:11.284943 19064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:11.285032 19064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "88b8d499a36c48389a7827642edd8e63"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "c7ced29cf851d6691ab400f3fc504d10"
server_key_iv: "5ba8a9537ffcc98ecc0c78ace90eea11"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.285099 19064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:11.297451 19064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:11.297775 19064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:11.297921 19064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:11.298144 19064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:11.298503 19064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:11.298542 19064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:11.298565 19064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:11.298579 19064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:11.304832 19064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43673
I20260928 04:52:11.304904 19191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43673 every 8 connection(s)
I20260928 04:52:11.305274 19064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:52:11.310444 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.305989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60953 (local address 127.0.98.62:44033)
0928 04:52:11.306029 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.306030 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.306274 (+ 244us) server_negotiation.cc:407] Connection header received
0928 04:52:11.306880 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.306882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.306894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.306936 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:11.307571 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.308045 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.308997 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.309239 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.309318 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.310169 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:11.310170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:11.310171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:11.310176 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:11.310191 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:11.310195 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:11.310249 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.310342 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.310370 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:52:11.310650 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.305984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.1:60953)
0928 04:52:11.306165 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:11.306195 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:11.306776 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:11.306996 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:11.306999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:11.307008 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:11.307475 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.307479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.308142 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.308143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.308896 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.308899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.309331 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.309332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.309452 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.310021 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:52:11.310022 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:11.310048 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:52:11.310068 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:11.310252 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:11.310256 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:11.310259 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:11.310343 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 04:52:11.310438 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260928 04:52:11.311133 19192 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44033
I20260928 04:52:11.311201 19192 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:11.311376 19192 heartbeater.cc:511] Master 127.0.98.62:44033 requested a full tablet report, sending...
I20260928 04:52:11.311856 19004 ts_manager.cc:194] Registered new tserver with Master: 88b8d499a36c48389a7827642edd8e63 (127.0.98.1:43673)
I20260928 04:52:11.312476 19004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60953
I20260928 04:52:11.314441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 19064
I20260928 04:52:11.314540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:52:11.314765 392 external_mini_cluster.cc:1428] Setting key ede4f8b6d27bfc43309e2ad9d67a673a
I20260928 04:52:11.315596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--builtin_ntp_servers=127.0.98.20:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:52:11.393426 19195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:11.393611 19195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:11.393631 19195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:11.394742 19195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:11.394783 19195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:11.394833 19195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:52:11.396128 19195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:11.396366 19195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:11.396651 19195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:11.401031 19200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:11.401038 19203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:11.401067 19201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:11.401369 19195 server_base.cc:1034] running on GCE node
I20260928 04:52:11.401588 19195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:11.401798 19195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:11.402928 19195 hybrid_clock.cc:648] HybridClock initialized: now 1790571131402889 us; error 56 us; skew 500 ppm
I20260928 04:52:11.407456 19195 webserver.cc:460] Webserver started at http://127.0.98.2:44499/ using document root <none> and password file <none>
I20260928 04:52:11.407696 19195 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:11.407742 19195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:11.407904 19195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:11.408896 19195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "69044ab63838466397a9a0927d4f0174"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "72ce16e56224e20d4203bfcc76e6fa43"
server_key_iv: "13758264ae3821b865c5db0c8750a51b"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.409281 19195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "69044ab63838466397a9a0927d4f0174"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "72ce16e56224e20d4203bfcc76e6fa43"
server_key_iv: "13758264ae3821b865c5db0c8750a51b"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.410535 19195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:52:11.411219 19209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:11.411391 19195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:11.411469 19195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "69044ab63838466397a9a0927d4f0174"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "72ce16e56224e20d4203bfcc76e6fa43"
server_key_iv: "13758264ae3821b865c5db0c8750a51b"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.411525 19195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:11.426296 19195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:11.426560 19195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:11.426728 19195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:11.426924 19195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:11.427213 19195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:11.427243 19195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:11.427302 19195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:11.427322 19195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:11.432878 19195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42117
I20260928 04:52:11.432950 19322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42117 every 8 connection(s)
I20260928 04:52:11.433334 19195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:52:11.437858 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.433985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35337 (local address 127.0.98.62:44033)
0928 04:52:11.434028 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.434030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.434270 (+ 240us) server_negotiation.cc:407] Connection header received
0928 04:52:11.434710 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.434711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.434720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.434776 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:11.435390 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.435852 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.436761 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.436918 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.436963 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.437660 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:11.437661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:11.437661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:11.437665 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:11.437678 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:11.437681 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:11.437727 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.437779 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.437803 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:52:11.438076 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.433968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.2:35337)
0928 04:52:11.434142 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:11.434180 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:11.434635 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:11.434830 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:11.434833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:11.434840 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:11.435315 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.435319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.435926 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.435927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.436668 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.436669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.436981 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.436982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.437077 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.437520 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:52:11.437520 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:11.437545 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:52:11.437564 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:11.437729 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:11.437732 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:11.437734 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:52:11.437774 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 04:52:11.437842 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260928 04:52:11.438606 19323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44033
I20260928 04:52:11.438714 19323 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:11.438886 19323 heartbeater.cc:511] Master 127.0.98.62:44033 requested a full tablet report, sending...
I20260928 04:52:11.439256 19004 ts_manager.cc:194] Registered new tserver with Master: 69044ab63838466397a9a0927d4f0174 (127.0.98.2:42117)
I20260928 04:52:11.439643 19004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35337
I20260928 04:52:11.440093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 19195
I20260928 04:52:11.440163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:52:11.440358 392 external_mini_cluster.cc:1428] Setting key 58e43ccf480ec827682995e65cccd069
I20260928 04:52:11.441124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--builtin_ntp_servers=127.0.98.20:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:52:11.517799 19326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:11.518004 19326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:11.518023 19326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:11.519001 19326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:11.519032 19326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:11.519083 19326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:52:11.520514 19326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:11.520699 19326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:11.520982 19326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:11.525743 19331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:11.525754 19334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:11.525764 19332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:11.525892 19326 server_base.cc:1034] running on GCE node
I20260928 04:52:11.526122 19326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:11.526366 19326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:11.527529 19326 hybrid_clock.cc:648] HybridClock initialized: now 1790571131527510 us; error 34 us; skew 500 ppm
I20260928 04:52:11.528648 19326 webserver.cc:460] Webserver started at http://127.0.98.3:46759/ using document root <none> and password file <none>
I20260928 04:52:11.528911 19326 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:11.528973 19326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:11.529096 19326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:11.530222 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "961b8bd9a18b45779e8cf51ab1dbea40"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "381ec1ed1bd9b1a3eaeb3ee5e53e4501"
server_key_iv: "139f56bdb0cc2a06e900942313a225f4"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.530593 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "961b8bd9a18b45779e8cf51ab1dbea40"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "381ec1ed1bd9b1a3eaeb3ee5e53e4501"
server_key_iv: "139f56bdb0cc2a06e900942313a225f4"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.531809 19326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:52:11.532498 19340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:11.532651 19326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:11.532748 19326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "961b8bd9a18b45779e8cf51ab1dbea40"
format_stamp: "Formatted at 2026-09-28 04:52:11 on dist-test-slave-jccs"
server_key: "381ec1ed1bd9b1a3eaeb3ee5e53e4501"
server_key_iv: "139f56bdb0cc2a06e900942313a225f4"
server_key_version: "encryptionkey@0"
I20260928 04:52:11.532824 19326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:11.547906 19326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:11.548167 19326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:11.548298 19326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:11.548498 19326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:11.548806 19326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:11.548858 19326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:11.548889 19326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:11.548915 19326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:11.554773 19326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40503
I20260928 04:52:11.554819 19453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40503 every 8 connection(s)
I20260928 04:52:11.555109 19326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:52:11.555584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 19326
I20260928 04:52:11.555680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:52:11.555917 392 external_mini_cluster.cc:1428] Setting key 1234ebc731f39b89c0c114cfcf146f2b
I20260928 04:52:11.559865 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.555859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55339 (local address 127.0.98.62:44033)
0928 04:52:11.555911 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.555913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.556149 (+ 236us) server_negotiation.cc:407] Connection header received
0928 04:52:11.556716 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.556717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.556730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.556785 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:11.557476 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.557929 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.558756 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.558914 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.558957 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.559688 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:11.559688 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:11.559689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:11.559692 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:11.559702 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:11.559704 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:11.559734 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.559794 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.559813 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 04:52:11.560071 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.555772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44033 (local address 127.0.98.3:55339)
0928 04:52:11.556009 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:11.556046 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:11.556628 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:11.556866 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:11.556870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:11.556879 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:11.557409 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.557413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.558002 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.558003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.558690 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.558691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.558979 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.558980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.559075 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.559580 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:52:11.559580 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:11.559604 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:52:11.559625 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:11.559742 (+ 117us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:11.559744 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:11.559746 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:52:11.559786 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 04:52:11.559850 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260928 04:52:11.560469 19454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44033
I20260928 04:52:11.560555 19454 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:11.560734 19454 heartbeater.cc:511] Master 127.0.98.62:44033 requested a full tablet report, sending...
I20260928 04:52:11.561097 19004 ts_manager.cc:194] Registered new tserver with Master: 961b8bd9a18b45779e8cf51ab1dbea40 (127.0.98.3:40503)
I20260928 04:52:11.561453 19004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55339
I20260928 04:52:11.567654 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:52:11.571583 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.568936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.0.98.62:44033)
0928 04:52:11.569008 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.569009 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.569058 (+ 49us) server_negotiation.cc:407] Connection header received
0928 04:52:11.569104 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.569104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.569113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.569161 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:11.569337 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.569788 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.570453 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.570613 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.570665 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.571307 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:11.571307 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:11.571308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:11.571311 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:11.571321 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:11.571324 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:11.571383 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.571458 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.571487 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 04:52:11.573295 19004 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:52:11.574780 19004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:52:11.580986 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.576881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40503 (local address 127.0.0.1:60018)
0928 04:52:11.577281 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:11.577293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:11.577364 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:11.577526 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:11.577529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:11.577532 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:11.577594 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.577596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.578860 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.578861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.580008 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.580011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.580794 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.580795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.580845 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.580854 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 04:52:11.580897 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 04:52:11.580932 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":7,"thread_start_us":88,"threads_started":1}
I20260928 04:52:11.581048 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.577099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60018 (local address 127.0.98.3:40503)
0928 04:52:11.577311 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.577315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.577373 (+ 58us) server_negotiation.cc:407] Connection header received
0928 04:52:11.577429 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.577431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.577443 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.577514 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:11.577657 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.578785 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.580078 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.580715 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.580791 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.580877 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.580925 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.580966 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
I20260928 04:52:11.581346 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.576875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43673 (local address 127.0.0.1:40712)
0928 04:52:11.577242 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:11.577251 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:11.577295 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:11.577888 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:11.577888 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:11.577890 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:11.577945 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.577947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.579073 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.579074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.579943 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.579945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.581097 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.581098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.581167 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.581198 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 04:52:11.581251 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 04:52:11.581288 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":6,"thread_start_us":55,"threads_started":1}
I20260928 04:52:11.581609 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.577047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40712 (local address 127.0.98.1:43673)
0928 04:52:11.577388 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.577392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.577402 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:52:11.577439 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.577441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.577454 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.577554 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:11.578014 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.578994 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.580023 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.580727 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.580791 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.580877 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.581497 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.581537 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":79,"threads_started":1}
I20260928 04:52:11.581797 19388 tablet_service.cc:1467] Processing CreateTablet for tablet 35eade9988584bafa1335c5751c90206 (DEFAULT_TABLE table=test-table [id=c33261fc5bc34479b896d5b5fbb9eba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:11.582187 19388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35eade9988584bafa1335c5751c90206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:11.582194 19126 tablet_service.cc:1467] Processing CreateTablet for tablet 35eade9988584bafa1335c5751c90206 (DEFAULT_TABLE table=test-table [id=c33261fc5bc34479b896d5b5fbb9eba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:11.582329 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.576875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42117 (local address 127.0.0.1:33346)
0928 04:52:11.577032 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:11.577067 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:11.577131 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:11.577831 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:11.577832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:11.577835 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:11.577874 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.577876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.578712 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.578713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.580008 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.580011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.582086 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.582087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.582148 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.582152 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:52:11.582217 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 04:52:11.582260 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":50,"spinlock_wait_cycles":1408,"thread_start_us":102,"threads_started":1}
I20260928 04:52:11.582485 19126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35eade9988584bafa1335c5751c90206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:11.582614 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.577055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.0.98.2:42117)
0928 04:52:11.577680 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.577684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.577692 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:52:11.577729 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.577731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.577745 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.577821 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:11.577954 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.578616 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.581148 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.581882 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.581962 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.582054 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.582231 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.582500 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":133,"threads_started":1}
I20260928 04:52:11.583175 19257 tablet_service.cc:1467] Processing CreateTablet for tablet 35eade9988584bafa1335c5751c90206 (DEFAULT_TABLE table=test-table [id=c33261fc5bc34479b896d5b5fbb9eba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:11.583461 19257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35eade9988584bafa1335c5751c90206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:11.585600 19468 tablet_bootstrap.cc:492] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63: Bootstrap starting.
I20260928 04:52:11.586169 19470 tablet_bootstrap.cc:492] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: Bootstrap starting.
I20260928 04:52:11.586458 19468 tablet_bootstrap.cc:654] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:11.586709 19470 tablet_bootstrap.cc:654] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:11.586798 19468 log.cc:824] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:11.587018 19470 log.cc:824] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:11.587138 19469 tablet_bootstrap.cc:492] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: Bootstrap starting.
I20260928 04:52:11.587531 19468 tablet_bootstrap.cc:492] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63: No bootstrap required, opened a new log
I20260928 04:52:11.587604 19468 ts_tablet_manager.cc:1397] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:52:11.587723 19470 tablet_bootstrap.cc:492] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: No bootstrap required, opened a new log
I20260928 04:52:11.587783 19470 ts_tablet_manager.cc:1397] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:52:11.589744 19469 tablet_bootstrap.cc:654] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:11.590035 19469 log.cc:824] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:11.590282 19470 raft_consensus.cc:348] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 } } peers { permanent_uuid: "88b8d499a36c48389a7827642edd8e63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43673 } } peers { permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 } }
I20260928 04:52:11.590282 19468 raft_consensus.cc:348] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 } } peers { permanent_uuid: "88b8d499a36c48389a7827642edd8e63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43673 } } peers { permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 } }
I20260928 04:52:11.590445 19470 raft_consensus.cc:374] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:11.590445 19468 raft_consensus.cc:374] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:11.590488 19468 raft_consensus.cc:729] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b8d499a36c48389a7827642edd8e63, State: Initialized, Role: FOLLOWER
I20260928 04:52:11.590488 19470 raft_consensus.cc:729] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69044ab63838466397a9a0927d4f0174, State: Initialized, Role: FOLLOWER
I20260928 04:52:11.590622 19470 consensus_queue.cc:260] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 } } peers { permanent_uuid: "88b8d499a36c48389a7827642edd8e63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43673 } } peers { permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 } }
I20260928 04:52:11.590617 19468 consensus_queue.cc:260] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 } } peers { permanent_uuid: "88b8d499a36c48389a7827642edd8e63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43673 } } peers { permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 } }
I20260928 04:52:11.590734 19469 tablet_bootstrap.cc:492] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: No bootstrap required, opened a new log
I20260928 04:52:11.590796 19469 ts_tablet_manager.cc:1397] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 04:52:11.590880 19468 ts_tablet_manager.cc:1428] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:52:11.590880 19470 ts_tablet_manager.cc:1428] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:52:11.590994 19192 heartbeater.cc:503] Master 127.0.98.62:44033 was elected leader, sending a full tablet report...
I20260928 04:52:11.591003 19323 heartbeater.cc:503] Master 127.0.98.62:44033 was elected leader, sending a full tablet report...
I20260928 04:52:11.593137 19469 raft_consensus.cc:348] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 } } peers { permanent_uuid: "88b8d499a36c48389a7827642edd8e63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43673 } } peers { permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 } }
I20260928 04:52:11.593341 19469 raft_consensus.cc:374] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:11.593391 19469 raft_consensus.cc:729] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961b8bd9a18b45779e8cf51ab1dbea40, State: Initialized, Role: FOLLOWER
I20260928 04:52:11.593499 19469 consensus_queue.cc:260] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 } } peers { permanent_uuid: "88b8d499a36c48389a7827642edd8e63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43673 } } peers { permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 } }
I20260928 04:52:11.593743 19469 ts_tablet_manager.cc:1428] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:52:11.593789 19454 heartbeater.cc:503] Master 127.0.98.62:44033 was elected leader, sending a full tablet report...
I20260928 04:52:11.664940 19475 raft_consensus.cc:482] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:52:11.665064 19475 raft_consensus.cc:504] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 } } peers { permanent_uuid: "88b8d499a36c48389a7827642edd8e63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43673 } } peers { permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 } }
I20260928 04:52:11.665513 19475 leader_election.cc:288] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69044ab63838466397a9a0927d4f0174 (127.0.98.2:42117), 961b8bd9a18b45779e8cf51ab1dbea40 (127.0.98.3:40503)
I20260928 04:52:11.668514 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.665749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58751 (local address 127.0.98.3:40503)
0928 04:52:11.665796 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.665798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.665808 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:52:11.665900 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.665901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.665911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.665971 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:11.666128 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.666705 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.667710 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.668200 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.668260 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.668268 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.668426 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.668461 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 04:52:11.668531 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.665682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40503 (local address 127.0.98.1:58751)
0928 04:52:11.665741 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:11.665753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:11.665802 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:11.665982 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:11.665983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:11.665985 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:11.666061 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.666063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.666791 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.666792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.667607 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.667609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.668280 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.668281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.668354 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.668358 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:52:11.668416 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 04:52:11.668457 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 04:52:11.668998 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.665696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42117 (local address 127.0.98.1:42371)
0928 04:52:11.665865 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:11.665874 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:11.665938 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:11.666080 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:11.666080 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:11.666082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:11.666176 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.666178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.667024 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.667025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.667948 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:11.667950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.668757 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.668758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:11.668831 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.668835 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:52:11.668892 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 04:52:11.668929 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260928 04:52:11.668977 19408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eade9988584bafa1335c5751c90206" candidate_uuid: "88b8d499a36c48389a7827642edd8e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" is_pre_election: true
I20260928 04:52:11.669030 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.665761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42371 (local address 127.0.98.2:42117)
0928 04:52:11.665802 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.665804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.665979 (+ 175us) server_negotiation.cc:407] Connection header received
0928 04:52:11.666009 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.666009 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.666019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.666068 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:11.666257 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.666946 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.668118 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.668652 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.668711 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.668720 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.668930 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.668974 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:52:11.669152 19408 raft_consensus.cc:2454] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b8d499a36c48389a7827642edd8e63 in term 0.
I20260928 04:52:11.669387 19081 leader_election.cc:302] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b8d499a36c48389a7827642edd8e63, 961b8bd9a18b45779e8cf51ab1dbea40; no voters:
I20260928 04:52:11.669423 19277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eade9988584bafa1335c5751c90206" candidate_uuid: "88b8d499a36c48389a7827642edd8e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69044ab63838466397a9a0927d4f0174" is_pre_election: true
I20260928 04:52:11.669540 19475 raft_consensus.cc:2781] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:52:11.669598 19475 raft_consensus.cc:482] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:52:11.669597 19277 raft_consensus.cc:2454] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b8d499a36c48389a7827642edd8e63 in term 0.
I20260928 04:52:11.669633 19475 raft_consensus.cc:3037] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:11.670361 19475 raft_consensus.cc:504] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 } } peers { permanent_uuid: "88b8d499a36c48389a7827642edd8e63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43673 } } peers { permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 } }
I20260928 04:52:11.670511 19475 leader_election.cc:288] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [CANDIDATE]: Term 1 election: Requested vote from peers 69044ab63838466397a9a0927d4f0174 (127.0.98.2:42117), 961b8bd9a18b45779e8cf51ab1dbea40 (127.0.98.3:40503)
I20260928 04:52:11.670660 19277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eade9988584bafa1335c5751c90206" candidate_uuid: "88b8d499a36c48389a7827642edd8e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69044ab63838466397a9a0927d4f0174"
I20260928 04:52:11.670692 19408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eade9988584bafa1335c5751c90206" candidate_uuid: "88b8d499a36c48389a7827642edd8e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "961b8bd9a18b45779e8cf51ab1dbea40"
I20260928 04:52:11.670753 19408 raft_consensus.cc:3037] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:11.670744 19277 raft_consensus.cc:3037] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:11.671639 19408 raft_consensus.cc:2454] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b8d499a36c48389a7827642edd8e63 in term 1.
I20260928 04:52:11.671643 19277 raft_consensus.cc:2454] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b8d499a36c48389a7827642edd8e63 in term 1.
I20260928 04:52:11.671789 19081 leader_election.cc:302] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b8d499a36c48389a7827642edd8e63, 961b8bd9a18b45779e8cf51ab1dbea40; no voters:
I20260928 04:52:11.671886 19475 raft_consensus.cc:2781] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:11.672043 19475 raft_consensus.cc:686] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 1 LEADER]: Becoming Leader. State: Replica: 88b8d499a36c48389a7827642edd8e63, State: Running, Role: LEADER
I20260928 04:52:11.672143 19475 consensus_queue.cc:237] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 } } peers { permanent_uuid: "88b8d499a36c48389a7827642edd8e63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43673 } } peers { permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 } }
I20260928 04:52:11.672930 19003 catalog_manager.cc:5495] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b8d499a36c48389a7827642edd8e63 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "88b8d499a36c48389a7827642edd8e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b8d499a36c48389a7827642edd8e63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 } health_report { overall_health: UNKNOWN } } }
W20260928 04:52:11.683990 19324 tablet.cc:2369] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:52:11.695540 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.693275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.0.98.1:43673)
0928 04:52:11.693327 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:11.693329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:11.693337 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:52:11.693376 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:11.693377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:11.693398 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:11.693482 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:11.693611 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.694066 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.694798 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:11.694994 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:11.695065 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:11.695196 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:11.695292 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:11.695396 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:11.695431 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 04:52:11.695442 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 04:52:11.697288 19277 raft_consensus.cc:1261] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174 [term 1 FOLLOWER]: Refusing update from remote peer 88b8d499a36c48389a7827642edd8e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:11.697335 19408 raft_consensus.cc:1261] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40 [term 1 FOLLOWER]: Refusing update from remote peer 88b8d499a36c48389a7827642edd8e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:11.697624 19478 consensus_queue.cc:1035] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69044ab63838466397a9a0927d4f0174" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:11.697786 19475 consensus_queue.cc:1035] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "961b8bd9a18b45779e8cf51ab1dbea40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:11.700824 19482 mvcc.cc:205] Tried to move back new op lower bound from 7334179355430064128 to 7334179355330523136. Current Snapshot: MvccSnapshot[applied={T|T < 7334179355430064128}]
I20260928 04:52:11.703022 19003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:11.703137 19003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:11.703881 19003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea8bee9ebfc043ddb7d8f1e3d4472f86: Sending DeleteTablet for 3 replicas of tablet 35eade9988584bafa1335c5751c90206
I20260928 04:52:11.704198 19126 tablet_service.cc:1514] Processing DeleteTablet for tablet 35eade9988584bafa1335c5751c90206 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:11 UTC) from {username='slave'} at 127.0.0.1:40712
I20260928 04:52:11.704195 19257 tablet_service.cc:1514] Processing DeleteTablet for tablet 35eade9988584bafa1335c5751c90206 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:11 UTC) from {username='slave'} at 127.0.0.1:33346
I20260928 04:52:11.704419 19492 tablet_replica.cc:330] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: stopping tablet replica
I20260928 04:52:11.704401 19388 tablet_service.cc:1514] Processing DeleteTablet for tablet 35eade9988584bafa1335c5751c90206 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:11 UTC) from {username='slave'} at 127.0.0.1:60018
I20260928 04:52:11.704514 19492 raft_consensus.cc:2229] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:52:11.704607 19492 raft_consensus.cc:2258] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:11.704603 19493 tablet_replica.cc:330] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: stopping tablet replica
I20260928 04:52:11.704749 19491 tablet_replica.cc:330] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63: stopping tablet replica
I20260928 04:52:11.704805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 19064
I20260928 04:52:11.704835 19491 raft_consensus.cc:2229] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:52:11.704931 19491 raft_consensus.cc:2258] T 35eade9988584bafa1335c5751c90206 P 88b8d499a36c48389a7827642edd8e63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:11.705042 19492 ts_tablet_manager.cc:1905] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:11.705513 19493 raft_consensus.cc:2229] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:52:11.705629 19493 raft_consensus.cc:2258] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:11.706705 19493 ts_tablet_manager.cc:1905] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:11.707335 19492 ts_tablet_manager.cc:1918] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:11.707410 19492 log.cc:1196] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/35eade9988584bafa1335c5751c90206
I20260928 04:52:11.707685 19492 ts_tablet_manager.cc:1939] T 35eade9988584bafa1335c5751c90206 P 69044ab63838466397a9a0927d4f0174: Deleting consensus metadata
I20260928 04:52:11.708153 18989 catalog_manager.cc:4841] TS 69044ab63838466397a9a0927d4f0174 (127.0.98.2:42117): tablet 35eade9988584bafa1335c5751c90206 (table test-table [id=c33261fc5bc34479b896d5b5fbb9eba3]) successfully deleted
I20260928 04:52:11.708228 19493 ts_tablet_manager.cc:1918] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:11.708299 19493 log.cc:1196] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/35eade9988584bafa1335c5751c90206
I20260928 04:52:11.708514 19493 ts_tablet_manager.cc:1939] T 35eade9988584bafa1335c5751c90206 P 961b8bd9a18b45779e8cf51ab1dbea40: Deleting consensus metadata
I20260928 04:52:11.708910 18990 catalog_manager.cc:4841] TS 961b8bd9a18b45779e8cf51ab1dbea40 (127.0.98.3:40503): tablet 35eade9988584bafa1335c5751c90206 (table test-table [id=c33261fc5bc34479b896d5b5fbb9eba3]) successfully deleted
W20260928 04:52:11.711262 18991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43673 (error 108)
I20260928 04:52:11.711721 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:11.711607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43673 (local address 127.0.0.1:40720)
0928 04:52:11.711661 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:11.711697 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 04:52:11.711889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 19195
W20260928 04:52:11.711880 18991 catalog_manager.cc:4584] TS 88b8d499a36c48389a7827642edd8e63 (127.0.98.1:43673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35eade9988584bafa1335c5751c90206: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43673: connect: Connection refused (error 111)
I20260928 04:52:11.716500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 19326
I20260928 04:52:11.721235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 18973
2026-09-28T04:52:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (723 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:11 dist-test-slave-jccs krb5kdc[19497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:11 dist-test-slave-jccs krb5kdc[19497](info): set up 2 sockets
Sep 28 04:52:11 dist-test-slave-jccs krb5kdc[19497](info): commencing operation
krb5kdc: starting...
W20260928 04:52:12.721046 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:12 dist-test-slave-jccs krb5kdc[19497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:12.834162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36203
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:12.912473 19510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:12.912729 19510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:12.912765 19510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:12.913936 19510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:12.914009 19510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:12.914036 19510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:12.914067 19510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:12.914090 19510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:12.915490 19510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:12.915756 19510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:12.915966 19510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:12.920611 19515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:12.920620 19518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:12.920634 19516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:12.920650 19510 server_base.cc:1034] running on GCE node
I20260928 04:52:12.920997 19510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:12.921320 19510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:12.922478 19510 hybrid_clock.cc:648] HybridClock initialized: now 1790571132922453 us; error 43 us; skew 500 ppm
Sep 28 04:52:12 dist-test-slave-jccs krb5kdc[19497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:12.924909 19510 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:12.925359 19510 webserver.cc:460] Webserver started at http://127.0.98.62:45769/ using document root <none> and password file <none>
I20260928 04:52:12.925581 19510 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:12.925624 19510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:12.925761 19510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:12.926744 19510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "52b59e480233446c8251512d185ac7e3"
format_stamp: "Formatted at 2026-09-28 04:52:12 on dist-test-slave-jccs"
server_key: "786516e4e4c4b073345fa6ac45a722e5"
server_key_iv: "053dffd6fbc84192ccf3fea1d053916d"
server_key_version: "encryptionkey@0"
I20260928 04:52:12.927080 19510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "52b59e480233446c8251512d185ac7e3"
format_stamp: "Formatted at 2026-09-28 04:52:12 on dist-test-slave-jccs"
server_key: "786516e4e4c4b073345fa6ac45a722e5"
server_key_iv: "053dffd6fbc84192ccf3fea1d053916d"
server_key_version: "encryptionkey@0"
I20260928 04:52:12.928294 19510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:52:12.929051 19525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:12.929246 19510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:52:12.929342 19510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "52b59e480233446c8251512d185ac7e3"
format_stamp: "Formatted at 2026-09-28 04:52:12 on dist-test-slave-jccs"
server_key: "786516e4e4c4b073345fa6ac45a722e5"
server_key_iv: "053dffd6fbc84192ccf3fea1d053916d"
server_key_version: "encryptionkey@0"
I20260928 04:52:12.929421 19510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:12.943581 19510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:12.945843 19510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:12.945992 19510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:12.950246 19510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36203
I20260928 04:52:12.950276 19577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36203 every 8 connection(s)
I20260928 04:52:12.950634 19510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:12.951508 19578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:12.953786 19578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3: Bootstrap starting.
I20260928 04:52:12.954347 19578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:12.954634 19578 log.cc:824] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:12.955297 19578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3: No bootstrap required, opened a new log
I20260928 04:52:12.956848 19578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b59e480233446c8251512d185ac7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260928 04:52:12.957002 19578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:12.957026 19578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b59e480233446c8251512d185ac7e3, State: Initialized, Role: FOLLOWER
I20260928 04:52:12.957252 19578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b59e480233446c8251512d185ac7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260928 04:52:12.957343 19578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:12.957384 19578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:12.957432 19578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:12.957998 19578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b59e480233446c8251512d185ac7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260928 04:52:12.958123 19578 leader_election.cc:302] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52b59e480233446c8251512d185ac7e3; no voters:
I20260928 04:52:12.958304 19578 leader_election.cc:288] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:12.958329 19581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:12.958505 19581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [term 1 LEADER]: Becoming Leader. State: Replica: 52b59e480233446c8251512d185ac7e3, State: Running, Role: LEADER
I20260928 04:52:12.958563 19578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:12.958596 19581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b59e480233446c8251512d185ac7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260928 04:52:12.958989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 19510
I20260928 04:52:12.958984 19582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52b59e480233446c8251512d185ac7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b59e480233446c8251512d185ac7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } } }
I20260928 04:52:12.959065 19582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:12.959127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:12.959112 19583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52b59e480233446c8251512d185ac7e3. Latest consensus state: current_term: 1 leader_uuid: "52b59e480233446c8251512d185ac7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b59e480233446c8251512d185ac7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } } }
I20260928 04:52:12.959172 19583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:12.959280 19588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:12.959355 392 external_mini_cluster.cc:1428] Setting key 524f3cceceee9a591e758c866f8d08cf
I20260928 04:52:12.959757 19588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:12.961519 19588 catalog_manager.cc:1347] Generated new cluster ID: 40dcef8207a3495f94c2403d61162e93
I20260928 04:52:12.961570 19588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:12.971743 19588 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:12.972429 19588 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 04:52:12 dist-test-slave-jccs krb5kdc[19497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:12.979151 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:12.960006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.0.98.62:36203)
0928 04:52:12.960193 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:12.960203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:12.960335 (+ 132us) server_negotiation.cc:407] Connection header received
0928 04:52:12.960492 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:12.960500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:12.960552 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:12.961256 (+ 704us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:12.969824 (+ 8568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:12.971205 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:12.972305 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:12.972912 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:12.973025 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:12.975754 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:12.975758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:12.975758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:12.975770 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:12.977656 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:12.978032 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:12.978034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:12.978035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:12.978128 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:12.978321 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:12.978322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:12.978322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:12.978559 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:12.978702 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:12.978805 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 04:52:12.978935 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260928 04:52:12.982856 19588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3: Generated new TSK 0
I20260928 04:52:12.983032 19588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 04:52:13.012105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:13.092556 19604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:13.092756 19604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:13.092778 19604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:13.093930 19604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:13.093982 19604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:13.094007 19604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:52:13.095359 19604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:13.095592 19604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:13.095801 19604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:13.100519 19612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:13.100519 19609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:13.100548 19610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:13.100754 19604 server_base.cc:1034] running on GCE node
I20260928 04:52:13.101027 19604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:13.101353 19604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:13.102535 19604 hybrid_clock.cc:648] HybridClock initialized: now 1790571133102520 us; error 33 us; skew 500 ppm
Sep 28 04:52:13 dist-test-slave-jccs krb5kdc[19497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:13.104820 19604 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 04:52:13.105288 19604 webserver.cc:460] Webserver started at http://127.0.98.1:42631/ using document root <none> and password file <none>
I20260928 04:52:13.105515 19604 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:13.105578 19604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:13.105708 19604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:13.106702 19604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "61d7b66f3d534d6ba07b03fd2c27485f"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "32aca45d1b46bead11f0b9c0421a3c43"
server_key_iv: "2c2a7101c2c15fb53d71c06de4f39efc"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.107075 19604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "61d7b66f3d534d6ba07b03fd2c27485f"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "32aca45d1b46bead11f0b9c0421a3c43"
server_key_iv: "2c2a7101c2c15fb53d71c06de4f39efc"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.108366 19604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:13.109165 19619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:13.109496 19604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:52:13.109594 19604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "61d7b66f3d534d6ba07b03fd2c27485f"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "32aca45d1b46bead11f0b9c0421a3c43"
server_key_iv: "2c2a7101c2c15fb53d71c06de4f39efc"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.109681 19604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:13.123358 19604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:13.125103 19604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:13.125284 19604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:13.125520 19604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:13.125840 19604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:13.125896 19604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:13.125932 19604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:13.125958 19604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:13.131513 19604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42039
I20260928 04:52:13.131583 19732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42039 every 8 connection(s)
I20260928 04:52:13.131939 19604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:52:13.136896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 19604
I20260928 04:52:13.137050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:52:13.137317 392 external_mini_cluster.cc:1428] Setting key 18868e77316c94873bda93ea68301669
Sep 28 04:52:13 dist-test-slave-jccs krb5kdc[19497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:13.141731 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.132706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57601 (local address 127.0.98.62:36203)
0928 04:52:13.132763 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.132765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.133003 (+ 238us) server_negotiation.cc:407] Connection header received
0928 04:52:13.133934 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.133936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.133946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.133997 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:13.134525 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.135023 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.135875 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.136046 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.136091 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.138490 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:13.138492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:13.138492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:13.138500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:13.140179 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:13.140708 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:13.140709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:13.140709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:13.140785 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:13.141042 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:13.141043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:13.141043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:13.141200 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:13.141263 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.141541 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.141639 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 04:52:13.141916 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.132705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:57601)
0928 04:52:13.132838 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:13.132906 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:13.133849 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:13.134051 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:13.134054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:13.134284 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:13.134459 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.134463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.135101 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.135102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.135810 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.135811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.136109 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.136110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.136197 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.136696 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:13.136720 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:13.138395 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:13.140245 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:13.140253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:13.140255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:13.140605 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:13.140859 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:13.140860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:13.140860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:13.140969 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:13.141255 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:13.141262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:13.141408 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 04:52:13.141519 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 04:52:13.141675 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260928 04:52:13.142491 19733 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36203
I20260928 04:52:13.142635 19733 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:13.142809 19733 heartbeater.cc:511] Master 127.0.98.62:36203 requested a full tablet report, sending...
I20260928 04:52:13.143345 19542 ts_manager.cc:194] Registered new tserver with Master: 61d7b66f3d534d6ba07b03fd2c27485f (127.0.98.1:42039)
I20260928 04:52:13.143975 19542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57601
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 04:52:13.161840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:13.238333 19738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:13.238513 19738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:13.238534 19738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:13.239579 19738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:13.239609 19738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:13.239632 19738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:52:13.240993 19738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:13.241153 19738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:13.241397 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:13.245975 19744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:13.245965 19743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:13.245965 19746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:13.246183 19738 server_base.cc:1034] running on GCE node
I20260928 04:52:13.246399 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:13.246592 19738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:13.247723 19738 hybrid_clock.cc:648] HybridClock initialized: now 1790571133247713 us; error 26 us; skew 500 ppm
Sep 28 04:52:13 dist-test-slave-jccs krb5kdc[19497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:13.250312 19738 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 04:52:13.250746 19738 webserver.cc:460] Webserver started at http://127.0.98.2:33797/ using document root <none> and password file <none>
I20260928 04:52:13.250965 19738 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:13.251006 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:13.251148 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:13.252170 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "e164f90031124b069341d0fd4cfd176c"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "0bd5d0f792005bb802ffda587ec033e3"
server_key_iv: "9dff51e94f37d51ecf544f5519a327c1"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.252544 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e164f90031124b069341d0fd4cfd176c"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "0bd5d0f792005bb802ffda587ec033e3"
server_key_iv: "9dff51e94f37d51ecf544f5519a327c1"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.253779 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 04:52:13.254458 19753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:13.254639 19738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:13.254726 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "e164f90031124b069341d0fd4cfd176c"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "0bd5d0f792005bb802ffda587ec033e3"
server_key_iv: "9dff51e94f37d51ecf544f5519a327c1"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.254796 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:13.275141 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:13.277467 19738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:13.277568 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:13.277746 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:13.278047 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:13.278100 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:13.278141 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:13.278163 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:13.283689 19738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33993
I20260928 04:52:13.283773 19866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33993 every 8 connection(s)
I20260928 04:52:13.284061 19738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:52:13.286221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 19738
I20260928 04:52:13.286365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:52:13.286571 392 external_mini_cluster.cc:1428] Setting key 21fffaddb82a719228d5f07254ea19c9
Sep 28 04:52:13 dist-test-slave-jccs krb5kdc[19497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:13.294263 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.284709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47961 (local address 127.0.98.62:36203)
0928 04:52:13.284789 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.284792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.284975 (+ 183us) server_negotiation.cc:407] Connection header received
0928 04:52:13.286081 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.286083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.286097 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.286171 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:13.286934 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.287598 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.288593 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.288858 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.288937 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.291452 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:13.291454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:13.291454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:13.291460 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:13.292857 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:13.293365 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:13.293365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:13.293366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:13.293414 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:13.293682 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:13.293682 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:13.293683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:13.293773 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:13.293817 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.294143 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.294203 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 04:52:13.294509 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.284661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:47961)
0928 04:52:13.284833 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:13.284866 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:13.285842 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:13.286237 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:13.286240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:13.286563 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:13.286849 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.286854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.287682 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.287683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.288507 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.288509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.288942 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.288944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.289072 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.289652 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:13.289668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:13.291360 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:13.292941 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:13.292947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:13.292949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:13.293288 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:13.293483 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:13.293484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:13.293484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:13.293616 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:13.293837 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:13.293843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:13.294000 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 04:52:13.294113 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 04:52:13.294283 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 04:52:13.295020 19867 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36203
I20260928 04:52:13.295133 19867 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:13.295279 19867 heartbeater.cc:511] Master 127.0.98.62:36203 requested a full tablet report, sending...
I20260928 04:52:13.295605 19542 ts_manager.cc:194] Registered new tserver with Master: e164f90031124b069341d0fd4cfd176c (127.0.98.2:33993)
I20260928 04:52:13.295982 19542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 04:52:13.309963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:13.387977 19872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:13.388201 19872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:13.388237 19872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:13.389461 19872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:13.389529 19872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:13.389573 19872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:52:13.391062 19872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:13.391335 19872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:13.391567 19872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:13.396458 19878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:13.396436 19880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:13.396432 19877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:13.396579 19872 server_base.cc:1034] running on GCE node
I20260928 04:52:13.396879 19872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:13.397115 19872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:13.398278 19872 hybrid_clock.cc:648] HybridClock initialized: now 1790571133398260 us; error 36 us; skew 500 ppm
Sep 28 04:52:13 dist-test-slave-jccs krb5kdc[19497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:13.400645 19872 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 04:52:13.401063 19872 webserver.cc:460] Webserver started at http://127.0.98.3:33643/ using document root <none> and password file <none>
I20260928 04:52:13.401353 19872 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:13.401423 19872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:13.401559 19872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:13.402633 19872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "de9152a759554433af1f5794b10f38a3"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "93f097d604dab1ce6e813726845752b5"
server_key_iv: "3fef42e02e17488ea02c8a31885d8fc8"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.402997 19872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "de9152a759554433af1f5794b10f38a3"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "93f097d604dab1ce6e813726845752b5"
server_key_iv: "3fef42e02e17488ea02c8a31885d8fc8"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.404321 19872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:13.404999 19887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:13.405151 19872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:13.405259 19872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "de9152a759554433af1f5794b10f38a3"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "93f097d604dab1ce6e813726845752b5"
server_key_iv: "3fef42e02e17488ea02c8a31885d8fc8"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.405341 19872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:13.418041 19872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:13.419708 19872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:13.419862 19872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:13.420073 19872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:13.420357 19872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:13.420413 19872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:13.420449 19872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:13.420483 19872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:13.426205 19872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45631
I20260928 04:52:13.426241 20000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45631 every 8 connection(s)
I20260928 04:52:13.426573 19872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 04:52:13 dist-test-slave-jccs krb5kdc[19497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:13.434373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 19872
I20260928 04:52:13.434481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:52:13.434665 392 external_mini_cluster.cc:1428] Setting key b9dabdfc2ef09be444ab1d0cae7d789f
I20260928 04:52:13.435542 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.427251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38711 (local address 127.0.98.62:36203)
0928 04:52:13.427321 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.427323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.427460 (+ 137us) server_negotiation.cc:407] Connection header received
0928 04:52:13.428274 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.428275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.428287 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.428334 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:13.428923 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.429497 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.430425 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.430592 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.430639 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.432723 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:13.432724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:13.432724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:13.432729 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:13.434123 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:13.434525 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:13.434526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:13.434527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:13.434582 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:13.434844 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:13.434845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:13.434845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:13.434945 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:13.435041 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.435381 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.435455 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:52:13.435738 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.427204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:38711)
0928 04:52:13.427369 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:13.427401 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:13.428189 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:13.428384 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:13.428387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:13.428652 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:13.428853 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.428857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.429569 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.429570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.430356 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.430358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.430657 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.430657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.430747 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.431182 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:13.431193 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:13.432641 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:13.434240 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:13.434244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:13.434246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:13.434428 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:13.434647 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:13.434648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:13.434648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:13.434772 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:13.435046 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:13.435053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:13.435225 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 04:52:13.435357 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 04:52:13.435506 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260928 04:52:13.436138 20001 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36203
I20260928 04:52:13.436230 20001 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:13.436358 20001 heartbeater.cc:511] Master 127.0.98.62:36203 requested a full tablet report, sending...
I20260928 04:52:13.436635 19542 ts_manager.cc:194] Registered new tserver with Master: de9152a759554433af1f5794b10f38a3 (127.0.98.3:45631)
I20260928 04:52:13.436954 19542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38711
I20260928 04:52:13.446288 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:52:13.453116 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.447609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.0.98.62:36203)
0928 04:52:13.447643 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.447645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.447773 (+ 128us) server_negotiation.cc:407] Connection header received
0928 04:52:13.447809 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.447809 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.447818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.447874 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:13.448367 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.448847 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.449672 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.449867 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.449928 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.450921 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:13.450922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:13.450922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:13.450927 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:13.452232 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:13.452530 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:13.452531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:13.452531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:13.452573 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:13.452721 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:13.452721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:13.452721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:13.452839 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:13.452907 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.452981 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.453029 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:52:13.454448 19542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:52:13.455968 19542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:52:13.462221 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.458014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45631 (local address 127.0.0.1:37100)
0928 04:52:13.458255 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:13.458291 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:13.458370 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:13.458885 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:13.458886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:13.458888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:13.458932 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.458933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.459906 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.459906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.461003 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.461006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.461906 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.461907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.461990 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.462004 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 04:52:13.462083 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 04:52:13.462143 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":51,"threads_started":1}
I20260928 04:52:13.462229 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.458105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.0.98.3:45631)
0928 04:52:13.458249 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.458254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.458614 (+ 360us) server_negotiation.cc:407] Connection header received
0928 04:52:13.458783 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.458785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.458802 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.458882 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:13.458990 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.459795 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.461085 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.461822 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.461893 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.461944 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.462105 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.462145 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260928 04:52:13.462558 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.458135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.0.98.1:42039)
0928 04:52:13.458438 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.458442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.458451 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:52:13.458486 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.458487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.458508 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.458591 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:13.459135 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.460307 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.461515 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.462097 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.462168 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.462221 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.462434 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.462474 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":75,"threads_started":1}
I20260928 04:52:13.462491 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.458017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42039 (local address 127.0.0.1:54778)
0928 04:52:13.458291 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:13.458302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:13.458358 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:13.458594 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:13.458597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:13.458599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:13.458685 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.458687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.460395 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.460396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.461431 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.461434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.462256 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.462257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.462332 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.462335 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:13.462403 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 04:52:13.462440 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":37,"thread_start_us":72,"threads_started":1}
I20260928 04:52:13.462632 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.458199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33993 (local address 127.0.0.1:39292)
0928 04:52:13.458675 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:13.458685 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:13.458766 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:13.459096 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:13.459097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:13.459099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:13.459147 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.459148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.460361 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.460362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.461436 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.461438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.462421 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.462423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.462478 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.462482 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:52:13.462527 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 04:52:13.462561 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":39,"threads_started":1}
I20260928 04:52:13.462826 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.458414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.0.98.2:33993)
0928 04:52:13.458596 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.458600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.458826 (+ 226us) server_negotiation.cc:407] Connection header received
0928 04:52:13.458867 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.458869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.458884 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.458985 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:13.459227 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.460248 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.461524 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.462229 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.462321 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.462375 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.462701 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.462760 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260928 04:52:13.463305 19667 tablet_service.cc:1467] Processing CreateTablet for tablet e19da8778a6a41bf8fee2f5a42f859ce (DEFAULT_TABLE table=test-table [id=e338b95d6e1443ecbe48d1fbd1fd2979]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:13.463380 19801 tablet_service.cc:1467] Processing CreateTablet for tablet e19da8778a6a41bf8fee2f5a42f859ce (DEFAULT_TABLE table=test-table [id=e338b95d6e1443ecbe48d1fbd1fd2979]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:13.463621 19801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e19da8778a6a41bf8fee2f5a42f859ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:13.463621 19667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e19da8778a6a41bf8fee2f5a42f859ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:13.466629 20015 tablet_bootstrap.cc:492] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f: Bootstrap starting.
I20260928 04:52:13.467145 19935 tablet_service.cc:1467] Processing CreateTablet for tablet e19da8778a6a41bf8fee2f5a42f859ce (DEFAULT_TABLE table=test-table [id=e338b95d6e1443ecbe48d1fbd1fd2979]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:13.467233 20015 tablet_bootstrap.cc:654] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:13.467438 19935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e19da8778a6a41bf8fee2f5a42f859ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:13.467756 20015 log.cc:824] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:13.468094 20016 tablet_bootstrap.cc:492] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: Bootstrap starting.
I20260928 04:52:13.468489 20015 tablet_bootstrap.cc:492] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f: No bootstrap required, opened a new log
I20260928 04:52:13.468552 20015 ts_tablet_manager.cc:1397] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:52:13.468741 20016 tablet_bootstrap.cc:654] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:13.469328 20016 log.cc:824] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:13.470100 20016 tablet_bootstrap.cc:492] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: No bootstrap required, opened a new log
I20260928 04:52:13.470165 20016 ts_tablet_manager.cc:1397] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:52:13.470659 20015 raft_consensus.cc:348] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 } } peers { permanent_uuid: "de9152a759554433af1f5794b10f38a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45631 } } peers { permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 } }
I20260928 04:52:13.470845 20015 raft_consensus.cc:374] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:13.470890 20015 raft_consensus.cc:729] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d7b66f3d534d6ba07b03fd2c27485f, State: Initialized, Role: FOLLOWER
I20260928 04:52:13.470999 20015 consensus_queue.cc:260] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 } } peers { permanent_uuid: "de9152a759554433af1f5794b10f38a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45631 } } peers { permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 } }
I20260928 04:52:13.471290 20015 ts_tablet_manager.cc:1428] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:52:13.471299 19733 heartbeater.cc:503] Master 127.0.98.62:36203 was elected leader, sending a full tablet report...
I20260928 04:52:13.472066 20016 raft_consensus.cc:348] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 } } peers { permanent_uuid: "de9152a759554433af1f5794b10f38a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45631 } } peers { permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 } }
I20260928 04:52:13.472213 20016 raft_consensus.cc:374] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:13.472258 20016 raft_consensus.cc:729] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e164f90031124b069341d0fd4cfd176c, State: Initialized, Role: FOLLOWER
I20260928 04:52:13.472380 20016 consensus_queue.cc:260] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 } } peers { permanent_uuid: "de9152a759554433af1f5794b10f38a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45631 } } peers { permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 } }
I20260928 04:52:13.472635 20016 ts_tablet_manager.cc:1428] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 04:52:13.472669 19867 heartbeater.cc:503] Master 127.0.98.62:36203 was elected leader, sending a full tablet report...
I20260928 04:52:13.472831 20019 tablet_bootstrap.cc:492] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: Bootstrap starting.
I20260928 04:52:13.473490 20019 tablet_bootstrap.cc:654] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:13.473945 20019 log.cc:824] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:13.474550 20019 tablet_bootstrap.cc:492] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: No bootstrap required, opened a new log
I20260928 04:52:13.474597 20019 ts_tablet_manager.cc:1397] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 04:52:13.475708 20019 raft_consensus.cc:348] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 } } peers { permanent_uuid: "de9152a759554433af1f5794b10f38a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45631 } } peers { permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 } }
I20260928 04:52:13.475829 20019 raft_consensus.cc:374] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:13.475849 20019 raft_consensus.cc:729] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9152a759554433af1f5794b10f38a3, State: Initialized, Role: FOLLOWER
I20260928 04:52:13.475911 20019 consensus_queue.cc:260] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 } } peers { permanent_uuid: "de9152a759554433af1f5794b10f38a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45631 } } peers { permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 } }
I20260928 04:52:13.476074 20019 ts_tablet_manager.cc:1428] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260928 04:52:13.476125 20001 heartbeater.cc:503] Master 127.0.98.62:36203 was elected leader, sending a full tablet report...
W20260928 04:52:13.534580 19868 tablet.cc:2369] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 04:52:13.632653 19734 tablet.cc:2369] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:52:13.639554 20023 raft_consensus.cc:482] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:52:13.639683 20023 raft_consensus.cc:504] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 } } peers { permanent_uuid: "de9152a759554433af1f5794b10f38a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45631 } } peers { permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 } }
I20260928 04:52:13.640030 20023 leader_election.cc:288] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61d7b66f3d534d6ba07b03fd2c27485f (127.0.98.1:42039), e164f90031124b069341d0fd4cfd176c (127.0.98.2:33993)
I20260928 04:52:13.643159 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.640296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49651 (local address 127.0.98.1:42039)
0928 04:52:13.640352 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.640355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.640364 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:52:13.640394 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.640395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.640408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.640487 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:13.640670 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.641327 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.642276 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.642812 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.642885 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.642895 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.643062 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.643097 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:52:13.643159 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.640227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42039 (local address 127.0.98.3:49651)
0928 04:52:13.640282 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:13.640294 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:13.640337 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:13.640512 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:13.640513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:13.640515 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:13.640599 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.640601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.641414 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.641415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.642167 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.642169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.642893 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.642894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.642976 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.642979 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:13.643043 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 04:52:13.643082 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 04:52:13.643241 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.640437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51747 (local address 127.0.98.2:33993)
0928 04:52:13.640483 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.640485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.640627 (+ 142us) server_negotiation.cc:407] Connection header received
0928 04:52:13.640655 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.640656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.640667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.640720 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:13.640876 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.641353 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.642370 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.642908 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.642967 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.642974 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.643134 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.643173 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:52:13.643251 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.640346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33993 (local address 127.0.98.3:51747)
0928 04:52:13.640527 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:13.640537 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:13.640593 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:13.640733 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:13.640734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:13.640736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:13.640796 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.640798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.641459 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.641460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.642292 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:13.642294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.642971 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.642972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:13.643058 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.643061 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:13.643128 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 04:52:13.643165 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260928 04:52:13.643630 19687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19da8778a6a41bf8fee2f5a42f859ce" candidate_uuid: "de9152a759554433af1f5794b10f38a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" is_pre_election: true
I20260928 04:52:13.643627 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19da8778a6a41bf8fee2f5a42f859ce" candidate_uuid: "de9152a759554433af1f5794b10f38a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e164f90031124b069341d0fd4cfd176c" is_pre_election: true
I20260928 04:52:13.643778 19821 raft_consensus.cc:2454] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de9152a759554433af1f5794b10f38a3 in term 0.
I20260928 04:52:13.643778 19687 raft_consensus.cc:2454] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de9152a759554433af1f5794b10f38a3 in term 0.
I20260928 04:52:13.643973 19889 leader_election.cc:302] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de9152a759554433af1f5794b10f38a3, e164f90031124b069341d0fd4cfd176c; no voters:
I20260928 04:52:13.644086 20023 raft_consensus.cc:2781] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:52:13.644174 20023 raft_consensus.cc:482] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:52:13.644210 20023 raft_consensus.cc:3037] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:13.644872 20023 raft_consensus.cc:504] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 } } peers { permanent_uuid: "de9152a759554433af1f5794b10f38a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45631 } } peers { permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 } }
I20260928 04:52:13.645004 20023 leader_election.cc:288] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [CANDIDATE]: Term 1 election: Requested vote from peers 61d7b66f3d534d6ba07b03fd2c27485f (127.0.98.1:42039), e164f90031124b069341d0fd4cfd176c (127.0.98.2:33993)
I20260928 04:52:13.645210 19687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19da8778a6a41bf8fee2f5a42f859ce" candidate_uuid: "de9152a759554433af1f5794b10f38a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d7b66f3d534d6ba07b03fd2c27485f"
I20260928 04:52:13.645246 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e19da8778a6a41bf8fee2f5a42f859ce" candidate_uuid: "de9152a759554433af1f5794b10f38a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e164f90031124b069341d0fd4cfd176c"
I20260928 04:52:13.645303 19687 raft_consensus.cc:3037] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:13.645337 19821 raft_consensus.cc:3037] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:13.645987 19687 raft_consensus.cc:2454] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de9152a759554433af1f5794b10f38a3 in term 1.
I20260928 04:52:13.646055 19821 raft_consensus.cc:2454] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de9152a759554433af1f5794b10f38a3 in term 1.
I20260928 04:52:13.646186 19889 leader_election.cc:302] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d7b66f3d534d6ba07b03fd2c27485f, de9152a759554433af1f5794b10f38a3; no voters:
I20260928 04:52:13.646299 20023 raft_consensus.cc:2781] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:13.646453 20023 raft_consensus.cc:686] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 1 LEADER]: Becoming Leader. State: Replica: de9152a759554433af1f5794b10f38a3, State: Running, Role: LEADER
I20260928 04:52:13.646554 20023 consensus_queue.cc:237] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 } } peers { permanent_uuid: "de9152a759554433af1f5794b10f38a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45631 } } peers { permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 } }
I20260928 04:52:13.647404 19541 catalog_manager.cc:5495] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to de9152a759554433af1f5794b10f38a3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "de9152a759554433af1f5794b10f38a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de9152a759554433af1f5794b10f38a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 } health_report { overall_health: UNKNOWN } } }
W20260928 04:52:13.677107 20002 tablet.cc:2369] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 04:52:13.687109 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.684953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.0.98.3:45631)
0928 04:52:13.685022 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.685025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.685034 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:52:13.685076 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.685077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.685093 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.685156 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:13.685298 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.685750 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.686439 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.686610 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.686665 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.686800 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:13.686889 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:13.686946 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.686978 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.686994 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 04:52:13.688789 19821 raft_consensus.cc:1261] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c [term 1 FOLLOWER]: Refusing update from remote peer de9152a759554433af1f5794b10f38a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:13.688825 19687 raft_consensus.cc:1261] T e19da8778a6a41bf8fee2f5a42f859ce P 61d7b66f3d534d6ba07b03fd2c27485f [term 1 FOLLOWER]: Refusing update from remote peer de9152a759554433af1f5794b10f38a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:13.689122 20023 consensus_queue.cc:1035] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e164f90031124b069341d0fd4cfd176c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:13.689301 20025 consensus_queue.cc:1035] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d7b66f3d534d6ba07b03fd2c27485f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:13.691684 20028 mvcc.cc:205] Tried to move back new op lower bound from 7334179363587268608 to 7334179363417436160. Current Snapshot: MvccSnapshot[applied={T|T < 7334179363587268608}]
I20260928 04:52:13.691834 20030 mvcc.cc:205] Tried to move back new op lower bound from 7334179363587268608 to 7334179363417436160. Current Snapshot: MvccSnapshot[applied={T|T < 7334179363587268608}]
I20260928 04:52:13.692152 20029 mvcc.cc:205] Tried to move back new op lower bound from 7334179363587268608 to 7334179363417436160. Current Snapshot: MvccSnapshot[applied={T|T < 7334179363587268608}]
I20260928 04:52:13.694088 19542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:13.694195 19542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:13.694897 19542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52b59e480233446c8251512d185ac7e3: Sending DeleteTablet for 3 replicas of tablet e19da8778a6a41bf8fee2f5a42f859ce
I20260928 04:52:13.695232 19935 tablet_service.cc:1514] Processing DeleteTablet for tablet e19da8778a6a41bf8fee2f5a42f859ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37100
I20260928 04:52:13.695284 19667 tablet_service.cc:1514] Processing DeleteTablet for tablet e19da8778a6a41bf8fee2f5a42f859ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54778
I20260928 04:52:13.695453 19801 tablet_service.cc:1514] Processing DeleteTablet for tablet e19da8778a6a41bf8fee2f5a42f859ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39292
I20260928 04:52:13.695483 20038 tablet_replica.cc:330] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: stopping tablet replica
I20260928 04:52:13.695572 20038 raft_consensus.cc:2229] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:52:13.695643 20040 tablet_replica.cc:330] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: stopping tablet replica
I20260928 04:52:13.695676 20038 raft_consensus.cc:2258] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:13.695729 20040 raft_consensus.cc:2229] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:52:13.695751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 19604
I20260928 04:52:13.695824 20040 raft_consensus.cc:2258] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:13.696321 20040 ts_tablet_manager.cc:1905] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:13.697190 20038 ts_tablet_manager.cc:1905] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:13.698431 20040 ts_tablet_manager.cc:1918] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:13.698494 20040 log.cc:1196] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/e19da8778a6a41bf8fee2f5a42f859ce
I20260928 04:52:13.698772 20040 ts_tablet_manager.cc:1939] T e19da8778a6a41bf8fee2f5a42f859ce P e164f90031124b069341d0fd4cfd176c: Deleting consensus metadata
I20260928 04:52:13.698817 20038 ts_tablet_manager.cc:1918] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:13.698895 20038 log.cc:1196] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/e19da8778a6a41bf8fee2f5a42f859ce
I20260928 04:52:13.699162 20038 ts_tablet_manager.cc:1939] T e19da8778a6a41bf8fee2f5a42f859ce P de9152a759554433af1f5794b10f38a3: Deleting consensus metadata
I20260928 04:52:13.699321 19527 catalog_manager.cc:4841] TS e164f90031124b069341d0fd4cfd176c (127.0.98.2:33993): tablet e19da8778a6a41bf8fee2f5a42f859ce (table test-table [id=e338b95d6e1443ecbe48d1fbd1fd2979]) successfully deleted
I20260928 04:52:13.699522 19526 catalog_manager.cc:4841] TS de9152a759554433af1f5794b10f38a3 (127.0.98.3:45631): tablet e19da8778a6a41bf8fee2f5a42f859ce (table test-table [id=e338b95d6e1443ecbe48d1fbd1fd2979]) successfully deleted
W20260928 04:52:13.702718 19527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42039 (error 108)
I20260928 04:52:13.703217 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.703072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42039 (local address 127.0.0.1:54786)
0928 04:52:13.703134 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:13.703183 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 04:52:13.703289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 19738
W20260928 04:52:13.705324 19527 catalog_manager.cc:4584] TS 61d7b66f3d534d6ba07b03fd2c27485f (127.0.98.1:42039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e19da8778a6a41bf8fee2f5a42f859ce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42039: connect: Connection refused (error 111)
I20260928 04:52:13.708500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 19872
I20260928 04:52:13.713387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 19510
2026-09-28T04:52:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1992 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T04:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:13Z Disabled control of system clock
I20260928 04:52:13.740067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43475
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:52:13.818429 20045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:13.818640 20045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:13.818662 20045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:13.819743 20045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:13.819778 20045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:13.819792 20045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:13.819804 20045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:13.819816 20045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:13.821056 20045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:13.821295 20045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:13.821525 20045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:13.826027 20050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:13.826035 20053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:13.826035 20051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:13.826397 20045 server_base.cc:1034] running on GCE node
I20260928 04:52:13.826643 20045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:13.826915 20045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:13.828080 20045 hybrid_clock.cc:648] HybridClock initialized: now 1790571133828075 us; error 35 us; skew 500 ppm
I20260928 04:52:13.829309 20045 webserver.cc:460] Webserver started at http://127.0.98.62:46781/ using document root <none> and password file <none>
I20260928 04:52:13.829533 20045 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:13.829598 20045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:13.829730 20045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:13.830735 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "0ab020f434b448c9b30d78f04ce87549"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "77cf1e089f876c25525a08de65a01468"
server_key_iv: "6aed8a4cdd9d3ace9ca37a68bf0d7f10"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.831094 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ab020f434b448c9b30d78f04ce87549"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "77cf1e089f876c25525a08de65a01468"
server_key_iv: "6aed8a4cdd9d3ace9ca37a68bf0d7f10"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.832299 20045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:13.832940 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:13.833110 20045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:13.833220 20045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "0ab020f434b448c9b30d78f04ce87549"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "77cf1e089f876c25525a08de65a01468"
server_key_iv: "6aed8a4cdd9d3ace9ca37a68bf0d7f10"
server_key_version: "encryptionkey@0"
I20260928 04:52:13.833303 20045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:13.850348 20045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:13.850620 20045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:13.850764 20045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:13.854573 20045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43475
I20260928 04:52:13.854605 20111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43475 every 8 connection(s)
I20260928 04:52:13.854902 20045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:13.855863 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:13.858050 20112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549: Bootstrap starting.
I20260928 04:52:13.858610 20112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:13.858844 20112 log.cc:824] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:13.859529 20112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549: No bootstrap required, opened a new log
I20260928 04:52:13.861011 20112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab020f434b448c9b30d78f04ce87549" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } }
I20260928 04:52:13.861140 20112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:13.861162 20112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab020f434b448c9b30d78f04ce87549, State: Initialized, Role: FOLLOWER
I20260928 04:52:13.861359 20112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab020f434b448c9b30d78f04ce87549" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } }
I20260928 04:52:13.861482 20112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:13.861508 20112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:13.861541 20112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:13.861984 20112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab020f434b448c9b30d78f04ce87549" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } }
I20260928 04:52:13.862062 20112 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab020f434b448c9b30d78f04ce87549; no voters:
I20260928 04:52:13.862197 20112 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:13.862267 20115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:13.862416 20115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [term 1 LEADER]: Becoming Leader. State: Replica: 0ab020f434b448c9b30d78f04ce87549, State: Running, Role: LEADER
I20260928 04:52:13.862454 20112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:13.862511 20115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab020f434b448c9b30d78f04ce87549" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } }
I20260928 04:52:13.862886 20117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab020f434b448c9b30d78f04ce87549. Latest consensus state: current_term: 1 leader_uuid: "0ab020f434b448c9b30d78f04ce87549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab020f434b448c9b30d78f04ce87549" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } } }
I20260928 04:52:13.862970 20116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ab020f434b448c9b30d78f04ce87549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab020f434b448c9b30d78f04ce87549" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43475 } } }
I20260928 04:52:13.862982 20117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:13.863044 20116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:13.863257 20123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:13.863788 20123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:13.864135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 20045
I20260928 04:52:13.864239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:13.864456 392 external_mini_cluster.cc:1428] Setting key 5de53422b5ad460f787022f44f8a3e42
I20260928 04:52:13.865414 20123 catalog_manager.cc:1347] Generated new cluster ID: aead1f5c911b4df99ded8df03f06dcfc
I20260928 04:52:13.865466 20123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:13.870503 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:13.864905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.0.98.62:43475)
0928 04:52:13.865078 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:13.865086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:13.865455 (+ 369us) server_negotiation.cc:407] Connection header received
0928 04:52:13.865569 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:13.865576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:13.865643 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:13.865775 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:13.865995 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.867322 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.868313 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:13.868962 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:13.869087 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:13.869820 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:13.869824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:13.869824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:13.870034 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:13.870074 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:13.870088 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:13.870158 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:13.870225 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 04:52:13.870301 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260928 04:52:13.893649 20123 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:13.894338 20123 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:13.904692 20123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549: Generated new TSK 0
I20260928 04:52:13.904878 20123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:13.909802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--builtin_ntp_servers=127.0.98.20:44825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 04:52:13.987504 20136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:13.987735 20136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:13.987771 20136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:13.988866 20136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:13.988924 20136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:13.988963 20136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 04:52:13.990252 20136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:13.990516 20136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:13.990789 20136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:13.995568 20141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:13.995579 20144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:13.995584 20142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:13.995814 20136 server_base.cc:1034] running on GCE node
I20260928 04:52:13.996007 20136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:13.996179 20136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:13.997335 20136 hybrid_clock.cc:648] HybridClock initialized: now 1790571133997315 us; error 30 us; skew 500 ppm
I20260928 04:52:13.998485 20136 webserver.cc:460] Webserver started at http://127.0.98.1:43551/ using document root <none> and password file <none>
I20260928 04:52:13.998725 20136 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:13.998795 20136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:13.998922 20136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:13.999903 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/data/instance:
uuid: "be8bd6a6801747109d5ffa1a0a2b3172"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "4ed536afa301bda39de8203ba30a9e88"
server_key_iv: "a939097a4592c6409a095c195614688d"
server_key_version: "encryptionkey@0"
I20260928 04:52:14.000255 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/wal/instance:
uuid: "be8bd6a6801747109d5ffa1a0a2b3172"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "4ed536afa301bda39de8203ba30a9e88"
server_key_iv: "a939097a4592c6409a095c195614688d"
server_key_version: "encryptionkey@0"
I20260928 04:52:14.001541 20136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:14.002228 20150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:14.002396 20136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:14.002490 20136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/wal
uuid: "be8bd6a6801747109d5ffa1a0a2b3172"
format_stamp: "Formatted at 2026-09-28 04:52:13 on dist-test-slave-jccs"
server_key: "4ed536afa301bda39de8203ba30a9e88"
server_key_iv: "a939097a4592c6409a095c195614688d"
server_key_version: "encryptionkey@0"
I20260928 04:52:14.002570 20136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:14.024256 20136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:14.024567 20136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:14.024712 20136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:14.024928 20136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:14.025313 20136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:14.025368 20136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:14.025406 20136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:14.025434 20136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:14.031095 20136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42197
I20260928 04:52:14.031137 20263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42197 every 8 connection(s)
I20260928 04:52:14.031458 20136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/data/info.pb
I20260928 04:52:14.034654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 20136
I20260928 04:52:14.034771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-0/wal/instance
I20260928 04:52:14.034968 392 external_mini_cluster.cc:1428] Setting key 64ff1c85892b9789b7c20a118920b4a2
I20260928 04:52:14.035993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--builtin_ntp_servers=127.0.98.20:44825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:52:14.038391 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.032143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39737 (local address 127.0.98.62:43475)
0928 04:52:14.032205 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:14.032208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:14.032413 (+ 205us) server_negotiation.cc:407] Connection header received
0928 04:52:14.032864 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:14.032865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:14.032879 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:14.032936 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:14.033725 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.034207 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.035350 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.035656 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.035728 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.038079 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:14.038080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:14.038081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:14.038086 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:14.038103 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:14.038108 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:14.038167 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:14.038242 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 04:52:14.038272 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:52:14.038491 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.032108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43475 (local address 127.0.98.1:39737)
0928 04:52:14.032293 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:14.032331 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:14.032783 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:14.033008 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:14.033012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:14.033019 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:14.033649 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.033653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.034287 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.034288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.035209 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.035211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.037224 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.037225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.037352 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.037940 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:52:14.037941 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:14.037969 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:52:14.037991 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:14.038172 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:14.038176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:14.038179 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:14.038230 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:52:14.038308 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260928 04:52:14.038959 20264 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43475
I20260928 04:52:14.039058 20264 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:14.039233 20264 heartbeater.cc:511] Master 127.0.98.62:43475 requested a full tablet report, sending...
I20260928 04:52:14.039778 20076 ts_manager.cc:194] Registered new tserver with Master: be8bd6a6801747109d5ffa1a0a2b3172 (127.0.98.1:42197)
I20260928 04:52:14.040755 20076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39737
W20260928 04:52:14.114100 20267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:14.114284 20267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:14.114303 20267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:14.115393 20267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:14.115424 20267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:14.115451 20267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 04:52:14.116653 20267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:14.116811 20267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:14.117015 20267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:14.121536 20273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:14.121542 20272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:14.121613 20267 server_base.cc:1034] running on GCE node
W20260928 04:52:14.121560 20275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:14.122004 20267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:14.122234 20267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:14.123368 20267 hybrid_clock.cc:648] HybridClock initialized: now 1790571134123346 us; error 38 us; skew 500 ppm
I20260928 04:52:14.124547 20267 webserver.cc:460] Webserver started at http://127.0.98.2:36003/ using document root <none> and password file <none>
I20260928 04:52:14.124765 20267 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:14.124810 20267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:14.124948 20267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:14.126015 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb453c65a608456fb9981ec34afaeb79"
format_stamp: "Formatted at 2026-09-28 04:52:14 on dist-test-slave-jccs"
server_key: "50ca0acdc3378d2eca694e2f5fed844a"
server_key_iv: "dc5847143c7af7f49ddf1e6a85f47086"
server_key_version: "encryptionkey@0"
I20260928 04:52:14.126386 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb453c65a608456fb9981ec34afaeb79"
format_stamp: "Formatted at 2026-09-28 04:52:14 on dist-test-slave-jccs"
server_key: "50ca0acdc3378d2eca694e2f5fed844a"
server_key_iv: "dc5847143c7af7f49ddf1e6a85f47086"
server_key_version: "encryptionkey@0"
I20260928 04:52:14.127621 20267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:52:14.128311 20281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:14.128458 20267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:52:14.128533 20267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/wal
uuid: "eb453c65a608456fb9981ec34afaeb79"
format_stamp: "Formatted at 2026-09-28 04:52:14 on dist-test-slave-jccs"
server_key: "50ca0acdc3378d2eca694e2f5fed844a"
server_key_iv: "dc5847143c7af7f49ddf1e6a85f47086"
server_key_version: "encryptionkey@0"
I20260928 04:52:14.128584 20267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:14.148880 20267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:14.149138 20267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:14.149292 20267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:14.149531 20267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:14.149832 20267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:14.149864 20267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:14.149885 20267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:14.149897 20267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:14.155225 20267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40389
I20260928 04:52:14.155299 20394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40389 every 8 connection(s)
I20260928 04:52:14.155622 20267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/data/info.pb
I20260928 04:52:14.160792 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.156454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46833 (local address 127.0.98.62:43475)
0928 04:52:14.156515 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:14.156517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:14.156761 (+ 244us) server_negotiation.cc:407] Connection header received
0928 04:52:14.157326 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:14.157327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:14.157339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:14.157391 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:14.158053 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.158538 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.159483 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.159686 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.159758 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.160507 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:14.160508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:14.160509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:14.160513 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:14.160527 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:14.160531 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:14.160594 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:14.160683 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 04:52:14.160712 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:52:14.161056 20397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.156446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43475 (local address 127.0.98.2:46833)
0928 04:52:14.156647 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:14.156679 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:14.157232 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:14.157470 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:14.157473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:14.157480 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:14.157955 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.157958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.158624 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.158625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.159360 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.159362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.159762 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.159763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.159888 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.160338 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:52:14.160339 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:14.160366 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:52:14.160386 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:14.160594 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:14.160600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:14.160603 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:14.160686 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 04:52:14.160805 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260928 04:52:14.161068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 20267
I20260928 04:52:14.161229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/wal/instance
I20260928 04:52:14.161504 392 external_mini_cluster.cc:1428] Setting key 7ae020e7e91da704e043640575c7ae60
I20260928 04:52:14.161600 20395 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43475
I20260928 04:52:14.161695 20395 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:14.161939 20395 heartbeater.cc:511] Master 127.0.98.62:43475 requested a full tablet report, sending...
I20260928 04:52:14.162314 20076 ts_manager.cc:194] Registered new tserver with Master: eb453c65a608456fb9981ec34afaeb79 (127.0.98.2:40389)
I20260928 04:52:14.162480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--builtin_ntp_servers=127.0.98.20:44825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 04:52:14.162761 20076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46833
W20260928 04:52:14.240643 20398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:14.240847 20398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:14.240868 20398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:14.242022 20398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:14.242060 20398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:14.242110 20398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 04:52:14.243396 20398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:14.243616 20398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:14.243885 20398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:14.248577 20404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:14.248555 20406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:14.248548 20403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:14.248816 20398 server_base.cc:1034] running on GCE node
I20260928 04:52:14.249007 20398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:14.249248 20398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:14.250398 20398 hybrid_clock.cc:648] HybridClock initialized: now 1790571134250392 us; error 31 us; skew 500 ppm
I20260928 04:52:14.251631 20398 webserver.cc:460] Webserver started at http://127.0.98.3:43759/ using document root <none> and password file <none>
I20260928 04:52:14.251861 20398 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:14.251905 20398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:14.252076 20398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:14.253090 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/data/instance:
uuid: "dafd291b709045aca5ac3cfe20899cb2"
format_stamp: "Formatted at 2026-09-28 04:52:14 on dist-test-slave-jccs"
server_key: "9997d6dcbd2a77047a9b3c35d236472b"
server_key_iv: "d248ad98b79e6459354746786f0eefc6"
server_key_version: "encryptionkey@0"
I20260928 04:52:14.253485 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dafd291b709045aca5ac3cfe20899cb2"
format_stamp: "Formatted at 2026-09-28 04:52:14 on dist-test-slave-jccs"
server_key: "9997d6dcbd2a77047a9b3c35d236472b"
server_key_iv: "d248ad98b79e6459354746786f0eefc6"
server_key_version: "encryptionkey@0"
I20260928 04:52:14.254760 20398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:52:14.255447 20412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:14.255620 20398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:14.255715 20398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/wal
uuid: "dafd291b709045aca5ac3cfe20899cb2"
format_stamp: "Formatted at 2026-09-28 04:52:14 on dist-test-slave-jccs"
server_key: "9997d6dcbd2a77047a9b3c35d236472b"
server_key_iv: "d248ad98b79e6459354746786f0eefc6"
server_key_version: "encryptionkey@0"
I20260928 04:52:14.255796 20398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:14.271786 20398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:14.272099 20398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:14.272240 20398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:14.272459 20398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 04:52:14.272837 20398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 04:52:14.272892 20398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:14.272930 20398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 04:52:14.272958 20398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:14.278498 20398 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35173
I20260928 04:52:14.278543 20525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35173 every 8 connection(s)
I20260928 04:52:14.278857 20398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/data/info.pb
I20260928 04:52:14.283353 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.279494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41081 (local address 127.0.98.62:43475)
0928 04:52:14.279558 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:14.279560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:14.279777 (+ 217us) server_negotiation.cc:407] Connection header received
0928 04:52:14.280199 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:14.280200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:14.280209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:14.280257 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:14.280847 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.281337 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.282325 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.282502 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.282544 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.283172 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:14.283173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:14.283173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:14.283177 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:14.283189 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:14.283193 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:14.283230 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:14.283278 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 04:52:14.283299 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:52:14.283520 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.279492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43475 (local address 127.0.98.3:41081)
0928 04:52:14.279665 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:14.279697 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:14.280126 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:14.280304 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:14.280307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:14.280314 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:14.280765 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.280768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.281408 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.281409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.282256 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.282258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.282551 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.282551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.282644 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.283066 (+ 422us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 04:52:14.283067 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:14.283091 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 04:52:14.283109 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:14.283229 (+ 120us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:14.283231 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:14.283233 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 04:52:14.283274 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 04:52:14.283345 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":98,"threads_started":1}
I20260928 04:52:14.284009 20526 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43475
I20260928 04:52:14.284109 20526 heartbeater.cc:464] Registering TS with master...
I20260928 04:52:14.284281 20526 heartbeater.cc:511] Master 127.0.98.62:43475 requested a full tablet report, sending...
I20260928 04:52:14.284654 20076 ts_manager.cc:194] Registered new tserver with Master: dafd291b709045aca5ac3cfe20899cb2 (127.0.98.3:35173)
I20260928 04:52:14.284993 20076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41081
I20260928 04:52:14.287452 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 20398
I20260928 04:52:14.287562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/wal/instance
I20260928 04:52:14.287766 392 external_mini_cluster.cc:1428] Setting key b3bdfcf697005d2e50b1161ff81c6d01
I20260928 04:52:14.288638 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 04:52:14.292577 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.289958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.0.98.62:43475)
0928 04:52:14.289996 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:14.289998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:14.290127 (+ 129us) server_negotiation.cc:407] Connection header received
0928 04:52:14.290148 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:14.290149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:14.290160 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:14.290209 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:14.290346 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.290776 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.291518 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.291684 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.291730 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.292328 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:14.292328 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:14.292329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:14.292332 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:14.292341 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:14.292345 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:14.292381 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:14.292444 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 04:52:14.292461 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:52:14.294142 20076 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 04:52:14.295644 20076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 04:52:14.301568 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.297608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40389 (local address 127.0.0.1:52722)
0928 04:52:14.297750 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:14.297786 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:14.297844 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:14.298325 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:14.298326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:14.298328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:14.298372 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.298374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.299274 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.299275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.300524 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.300526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.301268 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.301269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.301351 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.301366 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 04:52:14.301433 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 04:52:14.301483 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":23,"thread_start_us":47,"threads_started":1}
I20260928 04:52:14.301779 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.297621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35173 (local address 127.0.0.1:51928)
0928 04:52:14.297756 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:14.297788 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:14.297846 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:14.298238 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:14.298241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:14.298243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:14.298302 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.298303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.299560 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.299561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.300600 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.300603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.301425 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.301426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.301477 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.301480 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:14.301531 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:52:14.301560 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260928 04:52:14.301833 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.297974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.0.98.1:42197)
0928 04:52:14.298121 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:14.298125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:14.298138 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:52:14.298179 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:14.298180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:14.298212 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:14.298291 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:14.298518 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.299591 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.300574 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.301335 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.301409 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.301489 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:14.301721 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 04:52:14.301757 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260928 04:52:14.301925 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.297776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42197 (local address 127.0.0.1:56676)
0928 04:52:14.298000 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:14.298010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:14.298051 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:14.298404 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:14.298405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:14.298407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:14.298442 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.298443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.299674 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.299675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.300512 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.300514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.301527 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.301528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.301588 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.301592 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:52:14.301643 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 04:52:14.301683 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260928 04:52:14.302088 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.297712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.0.98.2:40389)
0928 04:52:14.298176 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:14.298180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:14.298189 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:52:14.298223 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:14.298225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:14.298238 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:14.298308 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:14.298449 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.299182 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.300621 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.301141 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.301265 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.301356 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:14.301766 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 04:52:14.301938 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":90,"threads_started":1}
I20260928 04:52:14.302230 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.297893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.0.98.3:35173)
0928 04:52:14.298057 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:14.298061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:14.298087 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:52:14.298122 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:14.298124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:14.298139 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:14.298217 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:14.298392 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.299478 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.300666 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.301347 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.301412 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.301489 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:14.301545 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 04:52:14.301579 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260928 04:52:14.302615 20198 tablet_service.cc:1467] Processing CreateTablet for tablet a8a226a7039e45df971995fca68a4e3e (DEFAULT_TABLE table=test-table [id=4d0270c5212c4809acbd97b08a2fac7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:14.302688 20329 tablet_service.cc:1467] Processing CreateTablet for tablet a8a226a7039e45df971995fca68a4e3e (DEFAULT_TABLE table=test-table [id=4d0270c5212c4809acbd97b08a2fac7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:14.302811 20460 tablet_service.cc:1467] Processing CreateTablet for tablet a8a226a7039e45df971995fca68a4e3e (DEFAULT_TABLE table=test-table [id=4d0270c5212c4809acbd97b08a2fac7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 04:52:14.302974 20198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a226a7039e45df971995fca68a4e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:14.302974 20329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a226a7039e45df971995fca68a4e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:14.303159 20460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a226a7039e45df971995fca68a4e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:14.305617 20540 tablet_bootstrap.cc:492] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: Bootstrap starting.
I20260928 04:52:14.306403 20540 tablet_bootstrap.cc:654] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:14.306680 20540 log.cc:824] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:14.306962 20542 tablet_bootstrap.cc:492] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172: Bootstrap starting.
I20260928 04:52:14.307288 20540 tablet_bootstrap.cc:492] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: No bootstrap required, opened a new log
I20260928 04:52:14.307356 20540 ts_tablet_manager.cc:1397] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 04:52:14.307526 20542 tablet_bootstrap.cc:654] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:14.307787 20542 log.cc:824] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:14.308051 20541 tablet_bootstrap.cc:492] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: Bootstrap starting.
I20260928 04:52:14.309518 20542 tablet_bootstrap.cc:492] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172: No bootstrap required, opened a new log
I20260928 04:52:14.309584 20542 ts_tablet_manager.cc:1397] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 04:52:14.309803 20540 raft_consensus.cc:348] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } } peers { permanent_uuid: "dafd291b709045aca5ac3cfe20899cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35173 } } peers { permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } }
I20260928 04:52:14.309973 20540 raft_consensus.cc:374] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:14.310010 20540 raft_consensus.cc:729] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dafd291b709045aca5ac3cfe20899cb2, State: Initialized, Role: FOLLOWER
I20260928 04:52:14.310145 20540 consensus_queue.cc:260] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } } peers { permanent_uuid: "dafd291b709045aca5ac3cfe20899cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35173 } } peers { permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } }
I20260928 04:52:14.310431 20540 ts_tablet_manager.cc:1428] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:52:14.310475 20526 heartbeater.cc:503] Master 127.0.98.62:43475 was elected leader, sending a full tablet report...
I20260928 04:52:14.311836 20542 raft_consensus.cc:348] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } } peers { permanent_uuid: "dafd291b709045aca5ac3cfe20899cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35173 } } peers { permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } }
I20260928 04:52:14.311955 20542 raft_consensus.cc:374] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:14.311987 20542 raft_consensus.cc:729] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8bd6a6801747109d5ffa1a0a2b3172, State: Initialized, Role: FOLLOWER
I20260928 04:52:14.312067 20542 consensus_queue.cc:260] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } } peers { permanent_uuid: "dafd291b709045aca5ac3cfe20899cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35173 } } peers { permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } }
I20260928 04:52:14.312394 20542 ts_tablet_manager.cc:1428] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 04:52:14.312439 20264 heartbeater.cc:503] Master 127.0.98.62:43475 was elected leader, sending a full tablet report...
I20260928 04:52:14.312964 20541 tablet_bootstrap.cc:654] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:14.313321 20541 log.cc:824] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:14.313890 20541 tablet_bootstrap.cc:492] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: No bootstrap required, opened a new log
I20260928 04:52:14.313939 20541 ts_tablet_manager.cc:1397] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 04:52:14.315565 20541 raft_consensus.cc:348] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } } peers { permanent_uuid: "dafd291b709045aca5ac3cfe20899cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35173 } } peers { permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } }
I20260928 04:52:14.315677 20541 raft_consensus.cc:374] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:14.315696 20541 raft_consensus.cc:729] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb453c65a608456fb9981ec34afaeb79, State: Initialized, Role: FOLLOWER
I20260928 04:52:14.315798 20541 consensus_queue.cc:260] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } } peers { permanent_uuid: "dafd291b709045aca5ac3cfe20899cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35173 } } peers { permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } }
I20260928 04:52:14.315990 20541 ts_tablet_manager.cc:1428] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 04:52:14.316071 20395 heartbeater.cc:503] Master 127.0.98.62:43475 was elected leader, sending a full tablet report...
I20260928 04:52:14.377292 20545 raft_consensus.cc:482] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 04:52:14.377406 20545 raft_consensus.cc:504] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } } peers { permanent_uuid: "dafd291b709045aca5ac3cfe20899cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35173 } } peers { permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } }
I20260928 04:52:14.377700 20545 leader_election.cc:288] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb453c65a608456fb9981ec34afaeb79 (127.0.98.2:40389), be8bd6a6801747109d5ffa1a0a2b3172 (127.0.98.1:42197)
I20260928 04:52:14.380440 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.377919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45935 (local address 127.0.98.2:40389)
0928 04:52:14.377977 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:14.377979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:14.377990 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:52:14.378071 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:14.378072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:14.378081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:14.378129 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:14.378327 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.378818 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.379755 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.380113 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.380163 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.380171 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:14.380328 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 04:52:14.380373 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:52:14.380435 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.377860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40389 (local address 127.0.98.3:45935)
0928 04:52:14.377919 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:14.377951 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:14.378005 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:14.378144 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:14.378145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:14.378146 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:14.378239 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.378241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.378889 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.378890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.379667 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.379669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.380182 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.380183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.380254 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.380257 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 04:52:14.380306 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:52:14.380339 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":18}
I20260928 04:52:14.380498 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.378031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57991 (local address 127.0.98.1:42197)
0928 04:52:14.378074 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:14.378076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:14.378208 (+ 132us) server_negotiation.cc:407] Connection header received
0928 04:52:14.378239 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:14.378239 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:14.378251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:14.378313 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 04:52:14.378447 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.378844 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.379794 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.380131 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.380216 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.380225 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:14.380388 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 04:52:14.380426 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:52:14.380528 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.377986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42197 (local address 127.0.98.3:57991)
0928 04:52:14.378128 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:14.378137 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:14.378183 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:14.378322 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:14.378323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:14.378325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 04:52:14.378385 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.378387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.378941 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.378942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.379680 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:14.379682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.380234 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.380234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:14.380312 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.380316 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 04:52:14.380372 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:52:14.380404 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260928 04:52:14.380781 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a226a7039e45df971995fca68a4e3e" candidate_uuid: "dafd291b709045aca5ac3cfe20899cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb453c65a608456fb9981ec34afaeb79" is_pre_election: true
I20260928 04:52:14.380803 20218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a226a7039e45df971995fca68a4e3e" candidate_uuid: "dafd291b709045aca5ac3cfe20899cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" is_pre_election: true
I20260928 04:52:14.380940 20349 raft_consensus.cc:2454] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dafd291b709045aca5ac3cfe20899cb2 in term 0.
I20260928 04:52:14.380940 20218 raft_consensus.cc:2454] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dafd291b709045aca5ac3cfe20899cb2 in term 0.
I20260928 04:52:14.381147 20416 leader_election.cc:302] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be8bd6a6801747109d5ffa1a0a2b3172, dafd291b709045aca5ac3cfe20899cb2; no voters:
I20260928 04:52:14.381338 20545 raft_consensus.cc:2781] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 04:52:14.381397 20545 raft_consensus.cc:482] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 04:52:14.381428 20545 raft_consensus.cc:3037] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:14.382107 20545 raft_consensus.cc:504] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } } peers { permanent_uuid: "dafd291b709045aca5ac3cfe20899cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35173 } } peers { permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } }
I20260928 04:52:14.382232 20545 leader_election.cc:288] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [CANDIDATE]: Term 1 election: Requested vote from peers eb453c65a608456fb9981ec34afaeb79 (127.0.98.2:40389), be8bd6a6801747109d5ffa1a0a2b3172 (127.0.98.1:42197)
I20260928 04:52:14.382395 20218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a226a7039e45df971995fca68a4e3e" candidate_uuid: "dafd291b709045aca5ac3cfe20899cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be8bd6a6801747109d5ffa1a0a2b3172"
I20260928 04:52:14.382416 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a226a7039e45df971995fca68a4e3e" candidate_uuid: "dafd291b709045aca5ac3cfe20899cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb453c65a608456fb9981ec34afaeb79"
I20260928 04:52:14.382475 20218 raft_consensus.cc:3037] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:14.382501 20349 raft_consensus.cc:3037] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:14.383164 20218 raft_consensus.cc:2454] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dafd291b709045aca5ac3cfe20899cb2 in term 1.
I20260928 04:52:14.383229 20349 raft_consensus.cc:2454] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dafd291b709045aca5ac3cfe20899cb2 in term 1.
I20260928 04:52:14.383320 20416 leader_election.cc:302] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be8bd6a6801747109d5ffa1a0a2b3172, dafd291b709045aca5ac3cfe20899cb2; no voters:
I20260928 04:52:14.383419 20545 raft_consensus.cc:2781] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:14.383585 20545 raft_consensus.cc:686] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 1 LEADER]: Becoming Leader. State: Replica: dafd291b709045aca5ac3cfe20899cb2, State: Running, Role: LEADER
I20260928 04:52:14.383689 20545 consensus_queue.cc:237] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } } peers { permanent_uuid: "dafd291b709045aca5ac3cfe20899cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35173 } } peers { permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } }
I20260928 04:52:14.384526 20075 catalog_manager.cc:5495] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dafd291b709045aca5ac3cfe20899cb2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dafd291b709045aca5ac3cfe20899cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dafd291b709045aca5ac3cfe20899cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } health_report { overall_health: UNKNOWN } } }
I20260928 04:52:14.396380 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.394197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.0.98.3:35173)
0928 04:52:14.394256 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:14.394258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:14.394266 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:52:14.394305 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:14.394306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:14.394320 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:14.394396 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:14.394518 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.394955 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.395677 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:14.395863 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:14.395936 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:14.396088 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:14.396175 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:14.396264 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:14.396282 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 04:52:14.396298 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 04:52:14.398034 20349 raft_consensus.cc:1261] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79 [term 1 FOLLOWER]: Refusing update from remote peer dafd291b709045aca5ac3cfe20899cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:14.398041 20218 raft_consensus.cc:1261] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172 [term 1 FOLLOWER]: Refusing update from remote peer dafd291b709045aca5ac3cfe20899cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:14.398375 20550 consensus_queue.cc:1035] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb453c65a608456fb9981ec34afaeb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:14.398506 20545 consensus_queue.cc:1035] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be8bd6a6801747109d5ffa1a0a2b3172" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:14.400952 20554 mvcc.cc:205] Tried to move back new op lower bound from 7334179366492188672 to 7334179366436724736. Current Snapshot: MvccSnapshot[applied={T|T < 7334179366492188672}]
I20260928 04:52:14.401356 20555 mvcc.cc:205] Tried to move back new op lower bound from 7334179366492188672 to 7334179366436724736. Current Snapshot: MvccSnapshot[applied={T|T < 7334179366492188672 or (T in {7334179366492188672})}]
I20260928 04:52:14.403303 20075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:14.403398 20075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 04:52:14.404083 20075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ab020f434b448c9b30d78f04ce87549: Sending DeleteTablet for 3 replicas of tablet a8a226a7039e45df971995fca68a4e3e
I20260928 04:52:14.404395 20460 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a226a7039e45df971995fca68a4e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:14 UTC) from {username='slave'} at 127.0.0.1:51928
I20260928 04:52:14.404513 20329 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a226a7039e45df971995fca68a4e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:14 UTC) from {username='slave'} at 127.0.0.1:52722
I20260928 04:52:14.404727 20564 tablet_replica.cc:330] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: stopping tablet replica
I20260928 04:52:14.404726 20198 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a226a7039e45df971995fca68a4e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 04:52:14 UTC) from {username='slave'} at 127.0.0.1:56676
I20260928 04:52:14.404850 20564 raft_consensus.cc:2229] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:52:14.404953 20565 tablet_replica.cc:330] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172: stopping tablet replica
I20260928 04:52:14.404959 20564 raft_consensus.cc:2258] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:14.405014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 20136
I20260928 04:52:14.405097 20565 raft_consensus.cc:2229] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 04:52:14.405234 20565 raft_consensus.cc:2258] T a8a226a7039e45df971995fca68a4e3e P be8bd6a6801747109d5ffa1a0a2b3172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:14.406112 20564 ts_tablet_manager.cc:1905] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:14.407949 20564 ts_tablet_manager.cc:1918] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:14.408021 20564 log.cc:1196] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-1/wal/wals/a8a226a7039e45df971995fca68a4e3e
I20260928 04:52:14.408316 20564 ts_tablet_manager.cc:1939] T a8a226a7039e45df971995fca68a4e3e P eb453c65a608456fb9981ec34afaeb79: Deleting consensus metadata
I20260928 04:52:14.408730 20063 catalog_manager.cc:4841] TS eb453c65a608456fb9981ec34afaeb79 (127.0.98.2:40389): tablet a8a226a7039e45df971995fca68a4e3e (table test-table [id=4d0270c5212c4809acbd97b08a2fac7e]) successfully deleted
W20260928 04:52:14.411988 20063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42197 (error 108)
I20260928 04:52:14.412004 20563 tablet_replica.cc:330] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: stopping tablet replica
I20260928 04:52:14.412143 20563 raft_consensus.cc:2229] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 04:52:14.412227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 20267
I20260928 04:52:14.412396 20563 raft_consensus.cc:2258] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 04:52:14.412443 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:14.412331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42197 (local address 127.0.0.1:56678)
0928 04:52:14.412375 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:14.412407 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 04:52:14.412545 20063 catalog_manager.cc:4584] TS be8bd6a6801747109d5ffa1a0a2b3172 (127.0.98.1:42197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8a226a7039e45df971995fca68a4e3e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42197: connect: Connection refused (error 111)
I20260928 04:52:14.413928 20563 ts_tablet_manager.cc:1905] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 04:52:14.415680 20563 ts_tablet_manager.cc:1918] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 04:52:14.415751 20563 log.cc:1196] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: Deleting WAL directory at /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790570897511889-392-0/minicluster-data/ts-2/wal/wals/a8a226a7039e45df971995fca68a4e3e
I20260928 04:52:14.415990 20563 ts_tablet_manager.cc:1939] T a8a226a7039e45df971995fca68a4e3e P dafd291b709045aca5ac3cfe20899cb2: Deleting consensus metadata
I20260928 04:52:14.416365 20062 catalog_manager.cc:4841] TS dafd291b709045aca5ac3cfe20899cb2 (127.0.98.3:35173): tablet a8a226a7039e45df971995fca68a4e3e (table test-table [id=4d0270c5212c4809acbd97b08a2fac7e]) successfully deleted
I20260928 04:52:14.417080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 20398
I20260928 04:52:14.422345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 20045
2026-09-28T04:52:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (708 ms)
[----------] 6 tests from EncryptionPolicyTest (8110 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:14 dist-test-slave-jccs krb5kdc[20569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:14 dist-test-slave-jccs krb5kdc[20569](info): set up 2 sockets
Sep 28 04:52:14 dist-test-slave-jccs krb5kdc[20569](info): commencing operation
krb5kdc: starting...
W20260928 04:52:15.424952 392 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:15 dist-test-slave-jccs krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:15.535331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:15.612893 20582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:15.613116 20582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:15.613160 20582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:15.614360 20582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:15.614427 20582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:15.614465 20582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:15.614496 20582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:15.614518 20582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:15.615896 20582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:15.616144 20582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:15.616339 20582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:15.620800 20587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:15.620920 20582 server_base.cc:1034] running on GCE node
W20260928 04:52:15.620785 20588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:15.621088 20590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:15.621476 20582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:15.621747 20582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:15.622903 20582 hybrid_clock.cc:648] HybridClock initialized: now 1790571135622899 us; error 35 us; skew 500 ppm
Sep 28 04:52:15 dist-test-slave-jccs krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:15.625288 20582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:15.625702 20582 webserver.cc:460] Webserver started at http://127.0.0.1:36865/ using document root <none> and password file <none>
I20260928 04:52:15.625867 20582 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:15.625905 20582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:15.625985 20582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:15.626891 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "1fef2ca9cd0d4707b62a1a380cc3794b"
format_stamp: "Formatted at 2026-09-28 04:52:15 on dist-test-slave-jccs"
server_key: "33c73108751c5d50fc1547467fce1d41"
server_key_iv: "65fe05a13b47b518696306f098062046"
server_key_version: "encryptionkey@0"
I20260928 04:52:15.627185 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fef2ca9cd0d4707b62a1a380cc3794b"
format_stamp: "Formatted at 2026-09-28 04:52:15 on dist-test-slave-jccs"
server_key: "33c73108751c5d50fc1547467fce1d41"
server_key_iv: "65fe05a13b47b518696306f098062046"
server_key_version: "encryptionkey@0"
I20260928 04:52:15.628381 20582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:52:15.629062 20597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:15.629244 20582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:15.629323 20582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "1fef2ca9cd0d4707b62a1a380cc3794b"
format_stamp: "Formatted at 2026-09-28 04:52:15 on dist-test-slave-jccs"
server_key: "33c73108751c5d50fc1547467fce1d41"
server_key_iv: "65fe05a13b47b518696306f098062046"
server_key_version: "encryptionkey@0"
I20260928 04:52:15.629372 20582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:15.639231 20582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:15.640969 20582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:15.641079 20582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:15.644747 20582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33079
I20260928 04:52:15.644809 20649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33079 every 8 connection(s)
I20260928 04:52:15.645107 20582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:15.646019 20650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:15.648171 20650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b: Bootstrap starting.
I20260928 04:52:15.648759 20650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:15.649051 20650 log.cc:824] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:15.649401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 20582
I20260928 04:52:15.649520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:15.649755 392 external_mini_cluster.cc:1428] Setting key 19ed1b225f36777ad63f6d6c55e4376b
I20260928 04:52:15.650179 20650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b: No bootstrap required, opened a new log
I20260928 04:52:15.651937 20650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fef2ca9cd0d4707b62a1a380cc3794b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } }
I20260928 04:52:15.652099 20650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:15.652144 20650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fef2ca9cd0d4707b62a1a380cc3794b, State: Initialized, Role: FOLLOWER
I20260928 04:52:15.652269 20650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fef2ca9cd0d4707b62a1a380cc3794b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } }
I20260928 04:52:15.652369 20650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:15.652410 20650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:15.652462 20650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:15.653342 20650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fef2ca9cd0d4707b62a1a380cc3794b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } }
I20260928 04:52:15.653451 20650 leader_election.cc:302] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fef2ca9cd0d4707b62a1a380cc3794b; no voters:
I20260928 04:52:15.653658 20650 leader_election.cc:288] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:15.653681 20655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:15.653789 20655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [term 1 LEADER]: Becoming Leader. State: Replica: 1fef2ca9cd0d4707b62a1a380cc3794b, State: Running, Role: LEADER
I20260928 04:52:15.653867 20655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fef2ca9cd0d4707b62a1a380cc3794b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } }
I20260928 04:52:15.653908 20650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:15.655300 20657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fef2ca9cd0d4707b62a1a380cc3794b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fef2ca9cd0d4707b62a1a380cc3794b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } } }
I20260928 04:52:15.655457 20657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:15.655593 20656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fef2ca9cd0d4707b62a1a380cc3794b. Latest consensus state: current_term: 1 leader_uuid: "1fef2ca9cd0d4707b62a1a380cc3794b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fef2ca9cd0d4707b62a1a380cc3794b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33079 } } }
I20260928 04:52:15.655867 20656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b [sys.catalog]: This master's current role is: LEADER
W20260928 04:52:15.655898 20670 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:15.655951 20670 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:15.656013 20671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:15.656174 20671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:15.657832 20671 catalog_manager.cc:1347] Generated new cluster ID: 2a2701a6f3f64b8da365053da5315b6f
I20260928 04:52:15.657884 20671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:52:15 dist-test-slave-jccs krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:52:15.662413 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:15.650459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42705 (local address 127.0.0.1:33079)
0928 04:52:15.650697 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:15.650705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:15.650729 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:52:15.651927 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:15.651934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:15.651984 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:15.652140 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:15.652859 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:15.654047 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:15.655095 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:15.655757 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:15.655864 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:15.658776 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:15.658780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:15.658780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:15.658791 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:15.660682 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:15.661130 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:15.661133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:15.661133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:15.661280 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:15.661533 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:15.661534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:15.661534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:15.661803 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:15.661955 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:15.662143 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 04:52:15.662236 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260928 04:52:15.663519 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:15.672238 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:15.665089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51485 (local address 127.0.0.1:33079)
0928 04:52:15.665118 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:15.665120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:15.665127 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:52:15.665236 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:15.665237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:15.665244 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:15.665295 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:15.665898 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:15.666604 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:15.667685 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:15.667998 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:15.668056 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:15.669451 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:15.669452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:15.669452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:15.669458 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:15.671015 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:15.671418 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:15.671419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:15.671419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:15.671481 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:15.671711 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:15.671711 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:15.671712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:15.671834 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:15.671926 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:15.672081 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 04:52:15.672156 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 04:52:15.679000 20671 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:15.679721 20671 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:15.690037 20671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fef2ca9cd0d4707b62a1a380cc3794b: Generated new TSK 0
I20260928 04:52:15.690243 20671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:15.692196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 20582
W20260928 04:52:15.696532 20567 connection.cc:535] client connection to 127.0.0.1:33079 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33079): Connection reset by peer (error 104)
2026-09-28T04:52:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1268 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:15 dist-test-slave-jccs krb5kdc[20681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:15 dist-test-slave-jccs krb5kdc[20681](info): set up 2 sockets
Sep 28 04:52:15 dist-test-slave-jccs krb5kdc[20681](info): commencing operation
krb5kdc: starting...
W20260928 04:52:16.684962 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:16 dist-test-slave-jccs krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:16.795287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:16.872888 20694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:16.873091 20694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:16.873126 20694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:16.874182 20694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:16.874248 20694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:16.874269 20694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:16.874289 20694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:16.874315 20694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:16.875676 20694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:16.875922 20694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:16.876144 20694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:16.880786 20699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:16.880947 20702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:16.880810 20694 server_base.cc:1034] running on GCE node
W20260928 04:52:16.880803 20700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:16.881354 20694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:16.881675 20694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:16.882819 20694 hybrid_clock.cc:648] HybridClock initialized: now 1790571136882799 us; error 36 us; skew 500 ppm
Sep 28 04:52:16 dist-test-slave-jccs krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:16.897742 20694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:16.898201 20694 webserver.cc:460] Webserver started at http://127.0.0.1:41509/ using document root <none> and password file <none>
I20260928 04:52:16.898443 20694 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:16.898487 20694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:16.898633 20694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:16.899745 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "30f1da968c624a83ac1835cb0089597a"
format_stamp: "Formatted at 2026-09-28 04:52:16 on dist-test-slave-jccs"
server_key: "ff73e4eaabafcb55af52d0fe11c69ad9"
server_key_iv: "2cc989f5c46621ebc300394e2e1eca9a"
server_key_version: "encryptionkey@0"
I20260928 04:52:16.900103 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "30f1da968c624a83ac1835cb0089597a"
format_stamp: "Formatted at 2026-09-28 04:52:16 on dist-test-slave-jccs"
server_key: "ff73e4eaabafcb55af52d0fe11c69ad9"
server_key_iv: "2cc989f5c46621ebc300394e2e1eca9a"
server_key_version: "encryptionkey@0"
I20260928 04:52:16.901441 20694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:52:16.902150 20709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:16.902331 20694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:52:16.902397 20694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "30f1da968c624a83ac1835cb0089597a"
format_stamp: "Formatted at 2026-09-28 04:52:16 on dist-test-slave-jccs"
server_key: "ff73e4eaabafcb55af52d0fe11c69ad9"
server_key_iv: "2cc989f5c46621ebc300394e2e1eca9a"
server_key_version: "encryptionkey@0"
I20260928 04:52:16.902493 20694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:16.919747 20694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:16.921645 20694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:16.921800 20694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:16.925432 20694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38261
I20260928 04:52:16.925483 20761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38261 every 8 connection(s)
I20260928 04:52:16.925843 20694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:16.926671 20762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:16.928890 20762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a: Bootstrap starting.
I20260928 04:52:16.929522 20762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:16.929805 20762 log.cc:824] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:16.929806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 20694
I20260928 04:52:16.929932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:16.930135 392 external_mini_cluster.cc:1428] Setting key d559cec08185e17f8578fad43becb0f3
I20260928 04:52:16.930555 20762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a: No bootstrap required, opened a new log
I20260928 04:52:16.932200 20762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1da968c624a83ac1835cb0089597a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } }
I20260928 04:52:16.932354 20762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:16.932397 20762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f1da968c624a83ac1835cb0089597a, State: Initialized, Role: FOLLOWER
I20260928 04:52:16.932521 20762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1da968c624a83ac1835cb0089597a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } }
I20260928 04:52:16.932618 20762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:16.932662 20762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:16.932729 20762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:16.933310 20762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1da968c624a83ac1835cb0089597a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } }
I20260928 04:52:16.933413 20762 leader_election.cc:302] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f1da968c624a83ac1835cb0089597a; no voters:
I20260928 04:52:16.933593 20762 leader_election.cc:288] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:16.933633 20767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:16.933751 20767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [term 1 LEADER]: Becoming Leader. State: Replica: 30f1da968c624a83ac1835cb0089597a, State: Running, Role: LEADER
I20260928 04:52:16.933823 20762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:16.933822 20767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1da968c624a83ac1835cb0089597a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } }
I20260928 04:52:16.934120 20767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f1da968c624a83ac1835cb0089597a. Latest consensus state: current_term: 1 leader_uuid: "30f1da968c624a83ac1835cb0089597a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1da968c624a83ac1835cb0089597a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } } }
I20260928 04:52:16.934201 20767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:16.934290 20768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f1da968c624a83ac1835cb0089597a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f1da968c624a83ac1835cb0089597a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38261 } } }
I20260928 04:52:16.934402 20768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a [sys.catalog]: This master's current role is: LEADER
W20260928 04:52:16.935575 20783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:16.935633 20783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:16.936097 20773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:16.936264 20773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:16.937392 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:16.930635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34129 (local address 127.0.0.1:38261)
0928 04:52:16.930818 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:16.930828 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:16.930862 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:52:16.932183 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:16.932189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:16.932234 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:16.932376 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:16.932588 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:16.934075 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:16.935195 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:16.935785 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:16.935900 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:16.936649 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:16.936653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:16.936653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:16.936858 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:16.936900 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:16.936914 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:16.936994 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:16.937157 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 04:52:16.937203 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260928 04:52:16.937698 20773 catalog_manager.cc:1347] Generated new cluster ID: a82795dbc4d340a0a85cf6fcf564ca59
I20260928 04:52:16.937748 20773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:16.938236 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:16.942288 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:16.939484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52717 (local address 127.0.0.1:38261)
0928 04:52:16.939530 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:16.939532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:16.939638 (+ 106us) server_negotiation.cc:407] Connection header received
0928 04:52:16.939727 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:16.939728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:16.939737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:16.939782 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:16.939916 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:16.940358 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:16.941056 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:16.941328 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:16.941407 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:16.942074 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:16.942075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:16.942076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:16.942080 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:16.942092 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:16.942096 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:16.942143 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:16.942195 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 04:52:16.942206 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:52:16.952544 20773 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:16.953022 20773 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:16.958186 20773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f1da968c624a83ac1835cb0089597a: Generated new TSK 0
I20260928 04:52:16.958351 20773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:16.961097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 20694
W20260928 04:52:16.964915 20679 connection.cc:535] client connection to 127.0.0.1:38261 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38261): Connection reset by peer (error 104)
2026-09-28T04:52:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1268 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:17 dist-test-slave-jccs krb5kdc[20793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:17 dist-test-slave-jccs krb5kdc[20793](info): set up 2 sockets
Sep 28 04:52:17 dist-test-slave-jccs krb5kdc[20793](info): commencing operation
krb5kdc: starting...
W20260928 04:52:17.973008 392 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:18 dist-test-slave-jccs krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:18.079355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37433
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:18.154065 20806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:18.154286 20806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:18.154320 20806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:18.155386 20806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:18.155483 20806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:18.155507 20806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:18.155534 20806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:18.155565 20806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:18.156970 20806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37433
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:18.157270 20806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:18.157485 20806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:18.162212 20814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:18.162221 20811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:18.162290 20806 server_base.cc:1034] running on GCE node
W20260928 04:52:18.162267 20812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:18.162632 20806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:18.162947 20806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:18.164103 20806 hybrid_clock.cc:648] HybridClock initialized: now 1790571138164084 us; error 32 us; skew 500 ppm
Sep 28 04:52:18 dist-test-slave-jccs krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:18.166456 20806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:18.166896 20806 webserver.cc:460] Webserver started at http://127.0.0.1:40739/ using document root <none> and password file <none>
I20260928 04:52:18.167121 20806 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:18.167188 20806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:18.167349 20806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:18.168349 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "df3ee04f680f4210bbaa158823d1b38a"
format_stamp: "Formatted at 2026-09-28 04:52:18 on dist-test-slave-jccs"
server_key: "166a9eb20ece38c4f2edbe9da02a600f"
server_key_iv: "e9cf1bb9ca02ef1062f1cd1e6c61afd4"
server_key_version: "encryptionkey@0"
I20260928 04:52:18.168691 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "df3ee04f680f4210bbaa158823d1b38a"
format_stamp: "Formatted at 2026-09-28 04:52:18 on dist-test-slave-jccs"
server_key: "166a9eb20ece38c4f2edbe9da02a600f"
server_key_iv: "e9cf1bb9ca02ef1062f1cd1e6c61afd4"
server_key_version: "encryptionkey@0"
I20260928 04:52:18.170030 20806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:52:18.170717 20821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:18.170879 20806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:18.170980 20806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "df3ee04f680f4210bbaa158823d1b38a"
format_stamp: "Formatted at 2026-09-28 04:52:18 on dist-test-slave-jccs"
server_key: "166a9eb20ece38c4f2edbe9da02a600f"
server_key_iv: "e9cf1bb9ca02ef1062f1cd1e6c61afd4"
server_key_version: "encryptionkey@0"
I20260928 04:52:18.171058 20806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:18.174208 20806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:18.175941 20806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:18.176087 20806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:18.179984 20806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37433
I20260928 04:52:18.180019 20873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37433 every 8 connection(s)
I20260928 04:52:18.180325 20806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:18.181262 20874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:18.183203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 20806
I20260928 04:52:18.183354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:18.183542 392 external_mini_cluster.cc:1428] Setting key 3c40b49824e412eed8c794b78a004a25
I20260928 04:52:18.183576 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a: Bootstrap starting.
I20260928 04:52:18.184257 20874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:18.184553 20874 log.cc:824] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:18.185286 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a: No bootstrap required, opened a new log
I20260928 04:52:18.185978 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:18.184039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33201 (local address 127.0.0.1:37433)
0928 04:52:18.184222 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:18.184229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:18.184252 (+ 23us) server_negotiation.cc:407] Connection header received
0928 04:52:18.184726 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:18.184732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:18.184758 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:18.184870 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:18.185575 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:18.185578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:18.185579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:18.185584 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:18.185624 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:18.185636 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:18.185731 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:18.185760 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 04:52:18.185798 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 04:52:18.186930 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:18.187661 20874 raft_consensus.cc:348] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3ee04f680f4210bbaa158823d1b38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } }
I20260928 04:52:18.187812 20874 raft_consensus.cc:374] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:18.187871 20874 raft_consensus.cc:729] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3ee04f680f4210bbaa158823d1b38a, State: Initialized, Role: FOLLOWER
I20260928 04:52:18.187997 20874 consensus_queue.cc:260] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3ee04f680f4210bbaa158823d1b38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } }
I20260928 04:52:18.188105 20874 raft_consensus.cc:388] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:18.188149 20874 raft_consensus.cc:482] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:18.188195 20874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:18.188714 20874 raft_consensus.cc:504] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3ee04f680f4210bbaa158823d1b38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } }
I20260928 04:52:18.188825 20874 leader_election.cc:302] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df3ee04f680f4210bbaa158823d1b38a; no voters:
I20260928 04:52:18.188992 20874 leader_election.cc:288] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:18.189013 20884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:18.189149 20884 raft_consensus.cc:686] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [term 1 LEADER]: Becoming Leader. State: Replica: df3ee04f680f4210bbaa158823d1b38a, State: Running, Role: LEADER
I20260928 04:52:18.189312 20874 sys_catalog.cc:564] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:18.189318 20884 consensus_queue.cc:237] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3ee04f680f4210bbaa158823d1b38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } }
I20260928 04:52:18.189743 20885 sys_catalog.cc:455] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df3ee04f680f4210bbaa158823d1b38a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3ee04f680f4210bbaa158823d1b38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } } }
I20260928 04:52:18.189862 20885 sys_catalog.cc:458] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:18.189743 20886 sys_catalog.cc:455] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df3ee04f680f4210bbaa158823d1b38a. Latest consensus state: current_term: 1 leader_uuid: "df3ee04f680f4210bbaa158823d1b38a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df3ee04f680f4210bbaa158823d1b38a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37433 } } }
I20260928 04:52:18.189991 20886 sys_catalog.cc:458] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:18.190145 20893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:18.190759 20893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:18.192080 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:18.188257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34119 (local address 127.0.0.1:37433)
0928 04:52:18.188295 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:18.188297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:18.190703 (+ 2406us) server_negotiation.cc:407] Connection header received
0928 04:52:18.190808 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:18.190809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:18.190814 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:18.190863 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:18.191868 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:18.191869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:18.191869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:18.191873 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:18.191888 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:18.191892 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:18.191948 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:18.192009 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 04:52:18.192021 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:52:18.193617 20893 catalog_manager.cc:1347] Generated new cluster ID: 035b6c4d991a4c0993a6c2cc656b20ff
I20260928 04:52:18.193674 20893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:18.209542 20893 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:18.209712 20893 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:18.213225 20893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df3ee04f680f4210bbaa158823d1b38a: Generated new TSK 0
I20260928 04:52:18.213395 20893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:18.217284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 20806
2026-09-28T04:52:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:18 dist-test-slave-jccs krb5kdc[20905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:18 dist-test-slave-jccs krb5kdc[20905](info): set up 2 sockets
Sep 28 04:52:18 dist-test-slave-jccs krb5kdc[20905](info): commencing operation
krb5kdc: starting...
W20260928 04:52:19.209106 392 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:19 dist-test-slave-jccs krb5kdc[20905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:19.316280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:19.393666 20918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:19.393842 20918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:19.393867 20918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:19.394868 20918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:19.394901 20918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:19.394913 20918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:19.394925 20918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:19.394936 20918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:19.396298 20918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:19.396481 20918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:19.396633 20918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 04:52:19.401310 20918 server_base.cc:1034] running on GCE node
W20260928 04:52:19.401293 20923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:19.401293 20926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:19.401311 20924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:19.401811 20918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:19.402105 20918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:19.403259 20918 hybrid_clock.cc:648] HybridClock initialized: now 1790571139403260 us; error 36 us; skew 500 ppm
Sep 28 04:52:19 dist-test-slave-jccs krb5kdc[20905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:19.405459 20918 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:19.405957 20918 webserver.cc:460] Webserver started at http://127.0.0.1:46097/ using document root <none> and password file <none>
I20260928 04:52:19.406193 20918 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:19.406272 20918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:19.406412 20918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:19.407377 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "1499a0549fb648c499e8c9fff82cafc5"
format_stamp: "Formatted at 2026-09-28 04:52:19 on dist-test-slave-jccs"
server_key: "aa4d5a9bcb36bc703500c3ee2e0009ee"
server_key_iv: "09dbd465fd6e66468f941c730d2d15f3"
server_key_version: "encryptionkey@0"
I20260928 04:52:19.407703 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "1499a0549fb648c499e8c9fff82cafc5"
format_stamp: "Formatted at 2026-09-28 04:52:19 on dist-test-slave-jccs"
server_key: "aa4d5a9bcb36bc703500c3ee2e0009ee"
server_key_iv: "09dbd465fd6e66468f941c730d2d15f3"
server_key_version: "encryptionkey@0"
I20260928 04:52:19.408921 20918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:52:19.409667 20933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:19.409823 20918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:19.409930 20918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "1499a0549fb648c499e8c9fff82cafc5"
format_stamp: "Formatted at 2026-09-28 04:52:19 on dist-test-slave-jccs"
server_key: "aa4d5a9bcb36bc703500c3ee2e0009ee"
server_key_iv: "09dbd465fd6e66468f941c730d2d15f3"
server_key_version: "encryptionkey@0"
I20260928 04:52:19.410006 20918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:19.426582 20918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:19.428195 20918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:19.428345 20918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:19.432207 20918 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36733
I20260928 04:52:19.432240 20985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36733 every 8 connection(s)
I20260928 04:52:19.432581 20918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:19.433410 20986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:19.435539 20986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5: Bootstrap starting.
I20260928 04:52:19.436029 20986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:19.436233 20986 log.cc:824] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:19.436780 20986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5: No bootstrap required, opened a new log
I20260928 04:52:19.438154 20986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499a0549fb648c499e8c9fff82cafc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36733 } }
I20260928 04:52:19.438272 20986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:19.438292 20986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1499a0549fb648c499e8c9fff82cafc5, State: Initialized, Role: FOLLOWER
I20260928 04:52:19.438412 20986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499a0549fb648c499e8c9fff82cafc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36733 } }
I20260928 04:52:19.438489 20986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:19.438510 20986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:19.438536 20986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:19.439014 20986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499a0549fb648c499e8c9fff82cafc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36733 } }
I20260928 04:52:19.439092 20986 leader_election.cc:302] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1499a0549fb648c499e8c9fff82cafc5; no voters:
I20260928 04:52:19.439203 20986 leader_election.cc:288] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:19.439291 20989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:19.439463 20986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:19.439476 20989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [term 1 LEADER]: Becoming Leader. State: Replica: 1499a0549fb648c499e8c9fff82cafc5, State: Running, Role: LEADER
I20260928 04:52:19.439574 20989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499a0549fb648c499e8c9fff82cafc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36733 } }
I20260928 04:52:19.439941 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1499a0549fb648c499e8c9fff82cafc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499a0549fb648c499e8c9fff82cafc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36733 } } }
I20260928 04:52:19.439986 20991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1499a0549fb648c499e8c9fff82cafc5. Latest consensus state: current_term: 1 leader_uuid: "1499a0549fb648c499e8c9fff82cafc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499a0549fb648c499e8c9fff82cafc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36733 } } }
I20260928 04:52:19.440097 20991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:19.440084 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:19.440439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 20918
I20260928 04:52:19.440490 20994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:19.440533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:19.440666 392 external_mini_cluster.cc:1428] Setting key 806770b1e11c965a1f2ae9c4042a23c4
I20260928 04:52:19.440974 20994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:19.442699 20994 catalog_manager.cc:1347] Generated new cluster ID: aaddbf8936f24511bd462c2b35bd268a
I20260928 04:52:19.442751 20994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:52:19 dist-test-slave-jccs krb5kdc[20905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:52:19.450408 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:19.441288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51329 (local address 127.0.0.1:36733)
0928 04:52:19.441437 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:19.441445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:19.441899 (+ 454us) server_negotiation.cc:407] Connection header received
0928 04:52:19.442040 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:19.442047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:19.442114 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:19.442959 (+ 845us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:19.443041 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:19.443875 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:19.444600 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:19.444998 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:19.445110 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:19.447347 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:19.447351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:19.447351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:19.447367 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:19.448863 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:19.449258 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:19.449260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:19.449261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:19.449358 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:19.449566 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:19.449567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:19.449567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:19.449796 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:19.449994 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:19.450063 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 04:52:19.450266 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260928 04:52:19.451200 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:19.452804 20994 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:19.453394 20994 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:19.457625 20994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1499a0549fb648c499e8c9fff82cafc5: Generated new TSK 0
I20260928 04:52:19.457768 20994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:19.458891 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:19.452395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49093 (local address 127.0.0.1:36733)
0928 04:52:19.452480 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:19.452481 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:19.452528 (+ 47us) server_negotiation.cc:407] Connection header received
0928 04:52:19.452598 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:19.452598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:19.452608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:19.452655 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:19.453242 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:19.453939 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:19.454838 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:19.455145 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:19.455211 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:19.456428 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:19.456429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:19.456429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:19.456436 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:19.457951 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:19.458230 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:19.458231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:19.458232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:19.458285 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:19.458426 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:19.458427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:19.458427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:19.458532 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:19.458622 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:19.458722 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 04:52:19.458808 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 04:52:19.460642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 20918
2026-09-28T04:52:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:19 dist-test-slave-jccs krb5kdc[21017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:19 dist-test-slave-jccs krb5kdc[21017](info): set up 2 sockets
Sep 28 04:52:19 dist-test-slave-jccs krb5kdc[21017](info): commencing operation
krb5kdc: starting...
W20260928 04:52:20.444909 392 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:20 dist-test-slave-jccs krb5kdc[21017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:20.556016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:20.636198 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:20.636446 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:20.636484 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:20.637712 21030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:20.637771 21030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:20.637795 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:20.637815 21030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:20.637846 21030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:20.639281 21030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:20.639551 21030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:20.639773 21030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:20.644397 21035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:20.644408 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:20.644445 21036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:20.644743 21030 server_base.cc:1034] running on GCE node
I20260928 04:52:20.644965 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:20.645308 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:20.646448 21030 hybrid_clock.cc:648] HybridClock initialized: now 1790571140646435 us; error 28 us; skew 500 ppm
Sep 28 04:52:20 dist-test-slave-jccs krb5kdc[21017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:20.648923 21030 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:20.649439 21030 webserver.cc:460] Webserver started at http://127.0.0.1:34877/ using document root <none> and password file <none>
I20260928 04:52:20.649659 21030 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:20.649719 21030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:20.649874 21030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:20.650894 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "a241c42bdeae4c72ae36c91a1f33ac01"
format_stamp: "Formatted at 2026-09-28 04:52:20 on dist-test-slave-jccs"
server_key: "c7dd3dad78ec136ecb3f60d661dc105c"
server_key_iv: "580161e4444fdd7b08fbf084b299fcaf"
server_key_version: "encryptionkey@0"
I20260928 04:52:20.651252 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "a241c42bdeae4c72ae36c91a1f33ac01"
format_stamp: "Formatted at 2026-09-28 04:52:20 on dist-test-slave-jccs"
server_key: "c7dd3dad78ec136ecb3f60d661dc105c"
server_key_iv: "580161e4444fdd7b08fbf084b299fcaf"
server_key_version: "encryptionkey@0"
I20260928 04:52:20.652544 21030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:20.653252 21045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:20.653441 21030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:20.653539 21030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "a241c42bdeae4c72ae36c91a1f33ac01"
format_stamp: "Formatted at 2026-09-28 04:52:20 on dist-test-slave-jccs"
server_key: "c7dd3dad78ec136ecb3f60d661dc105c"
server_key_iv: "580161e4444fdd7b08fbf084b299fcaf"
server_key_version: "encryptionkey@0"
I20260928 04:52:20.653630 21030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:20.665246 21030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:20.669111 21030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:20.669327 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:20.673344 21030 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36031
I20260928 04:52:20.673386 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36031 every 8 connection(s)
I20260928 04:52:20.673755 21030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:20.674583 21098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:20.676797 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01: Bootstrap starting.
I20260928 04:52:20.677462 21098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:20.677735 21098 log.cc:824] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:20.678426 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01: No bootstrap required, opened a new log
I20260928 04:52:20.679996 21098 raft_consensus.cc:348] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241c42bdeae4c72ae36c91a1f33ac01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } }
I20260928 04:52:20.680136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 21030
I20260928 04:52:20.680154 21098 raft_consensus.cc:374] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:20.680207 21098 raft_consensus.cc:729] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a241c42bdeae4c72ae36c91a1f33ac01, State: Initialized, Role: FOLLOWER
I20260928 04:52:20.680253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:20.680434 392 external_mini_cluster.cc:1428] Setting key edf7178752c63944e1154afc4bf63a76
I20260928 04:52:20.680389 21098 consensus_queue.cc:260] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241c42bdeae4c72ae36c91a1f33ac01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } }
I20260928 04:52:20.680485 21098 raft_consensus.cc:388] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:20.680531 21098 raft_consensus.cc:482] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:20.680581 21098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:20.681356 21098 raft_consensus.cc:504] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241c42bdeae4c72ae36c91a1f33ac01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } }
I20260928 04:52:20.681473 21098 leader_election.cc:302] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a241c42bdeae4c72ae36c91a1f33ac01; no voters:
I20260928 04:52:20.681628 21098 leader_election.cc:288] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:20.681655 21103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:20.681797 21103 raft_consensus.cc:686] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [term 1 LEADER]: Becoming Leader. State: Replica: a241c42bdeae4c72ae36c91a1f33ac01, State: Running, Role: LEADER
I20260928 04:52:20.681928 21103 consensus_queue.cc:237] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241c42bdeae4c72ae36c91a1f33ac01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } }
I20260928 04:52:20.681921 21098 sys_catalog.cc:564] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:20.682260 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a241c42bdeae4c72ae36c91a1f33ac01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241c42bdeae4c72ae36c91a1f33ac01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } } }
I20260928 04:52:20.682576 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:20.682384 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a241c42bdeae4c72ae36c91a1f33ac01. Latest consensus state: current_term: 1 leader_uuid: "a241c42bdeae4c72ae36c91a1f33ac01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a241c42bdeae4c72ae36c91a1f33ac01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } } }
I20260928 04:52:20.682817 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:20.683576 21108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:20.684053 21108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:20.685701 21108 catalog_manager.cc:1347] Generated new cluster ID: 9da8209d0d084fee972c709017b9f213
I20260928 04:52:20.685750 21108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:20.690214 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:20.680947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48671 (local address 127.0.0.1:36031)
0928 04:52:20.681517 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:20.681533 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:20.681582 (+ 49us) server_negotiation.cc:407] Connection header received
0928 04:52:20.681716 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:20.681730 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:20.681791 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:20.681966 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:20.682185 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:20.683389 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:20.684441 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:20.685052 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:20.685534 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:20.689494 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:20.689500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:20.689501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:20.689747 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:20.689776 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:20.689788 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:20.689861 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:20.689940 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 04:52:20.690003 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":485,"threads_started":1}
I20260928 04:52:20.691089 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:20.695291 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:20.692405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55913 (local address 127.0.0.1:36031)
0928 04:52:20.692440 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:20.692442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:20.692554 (+ 112us) server_negotiation.cc:407] Connection header received
0928 04:52:20.692625 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:20.692626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:20.692636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:20.692681 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:20.692839 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:20.693396 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:20.694093 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:20.694296 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:20.694380 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:20.695026 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:20.695027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:20.695027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:20.695033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:20.695045 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:20.695049 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:20.695102 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:20.695165 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 04:52:20.695188 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:52:20.714542 21108 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:20.715094 21108 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:20.721474 21108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a241c42bdeae4c72ae36c91a1f33ac01: Generated new TSK 0
I20260928 04:52:20.721681 21108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:20.728976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 21030
2026-09-28T04:52:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1268 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:20 dist-test-slave-jccs krb5kdc[21129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:20 dist-test-slave-jccs krb5kdc[21129](info): set up 2 sockets
Sep 28 04:52:20 dist-test-slave-jccs krb5kdc[21129](info): commencing operation
krb5kdc: starting...
W20260928 04:52:21.749207 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:21 dist-test-slave-jccs krb5kdc[21129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:21.863981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:21.942802 21142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:21.943053 21142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:21.943091 21142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:21.944238 21142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:21.944307 21142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:21.944332 21142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:21.944353 21142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:21.944382 21142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:21.945933 21142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:21.946208 21142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:21.946431 21142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:21.951226 21148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:21.951248 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:21.951299 21142 server_base.cc:1034] running on GCE node
W20260928 04:52:21.951251 21150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:21.951593 21142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:21.951834 21142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:21.953011 21142 hybrid_clock.cc:648] HybridClock initialized: now 1790571141952991 us; error 37 us; skew 500 ppm
Sep 28 04:52:21 dist-test-slave-jccs krb5kdc[21129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:21.962141 21142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:21.962596 21142 webserver.cc:460] Webserver started at http://127.0.0.1:37089/ using document root <none> and password file <none>
I20260928 04:52:21.962836 21142 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:21.962905 21142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:21.963053 21142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:21.964114 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "d8a66861a15f447b8e9edf8691b9ef9e"
format_stamp: "Formatted at 2026-09-28 04:52:21 on dist-test-slave-jccs"
server_key: "bf6b9c76b78c9bb0cea8bc3c0d425252"
server_key_iv: "b7e4a65f5604c8a90c6949a52fc563e2"
server_key_version: "encryptionkey@0"
I20260928 04:52:21.964479 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "d8a66861a15f447b8e9edf8691b9ef9e"
format_stamp: "Formatted at 2026-09-28 04:52:21 on dist-test-slave-jccs"
server_key: "bf6b9c76b78c9bb0cea8bc3c0d425252"
server_key_iv: "b7e4a65f5604c8a90c6949a52fc563e2"
server_key_version: "encryptionkey@0"
I20260928 04:52:21.965890 21142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:21.966619 21157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:21.966789 21142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:21.966892 21142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "d8a66861a15f447b8e9edf8691b9ef9e"
format_stamp: "Formatted at 2026-09-28 04:52:21 on dist-test-slave-jccs"
server_key: "bf6b9c76b78c9bb0cea8bc3c0d425252"
server_key_iv: "b7e4a65f5604c8a90c6949a52fc563e2"
server_key_version: "encryptionkey@0"
I20260928 04:52:21.966965 21142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:21.970367 21142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:21.972270 21142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:21.972425 21142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:21.976167 21142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46101
I20260928 04:52:21.976192 21209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46101 every 8 connection(s)
I20260928 04:52:21.976516 21142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:21.977517 21210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:21.978276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 21142
I20260928 04:52:21.978395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:21.978585 392 external_mini_cluster.cc:1428] Setting key 9541b65c9da6b19ae482961627687878
I20260928 04:52:21.980590 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e: Bootstrap starting.
I20260928 04:52:21.980845 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:21.979098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33665 (local address 127.0.0.1:46101)
0928 04:52:21.979269 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:21.979276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:21.979301 (+ 25us) server_negotiation.cc:407] Connection header received
0928 04:52:21.979680 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:21.979686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:21.979710 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:21.979814 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:21.980412 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:21.980416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:21.980416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:21.980420 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:21.980452 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:21.980462 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:21.980514 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:21.980602 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 04:52:21.980652 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260928 04:52:21.981328 21210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:21.981595 21210 log.cc:824] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:21.981679 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:21.982198 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e: No bootstrap required, opened a new log
I20260928 04:52:21.983954 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:21.983081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50017 (local address 127.0.0.1:46101)
0928 04:52:21.983117 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:21.983119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:21.983208 (+ 89us) server_negotiation.cc:407] Connection header received
0928 04:52:21.983270 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:21.983271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:21.983274 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:21.983317 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:21.983763 (+ 446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:21.983764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:21.983764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:21.983767 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:21.983777 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:21.983780 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:21.983817 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:21.983877 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 04:52:21.983886 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:52:21.984479 21210 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a66861a15f447b8e9edf8691b9ef9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260928 04:52:21.984660 21210 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:21.984705 21210 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a66861a15f447b8e9edf8691b9ef9e, State: Initialized, Role: FOLLOWER
I20260928 04:52:21.984822 21210 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a66861a15f447b8e9edf8691b9ef9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260928 04:52:21.984925 21210 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:21.984969 21210 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:21.985018 21210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:21.985594 21210 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a66861a15f447b8e9edf8691b9ef9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260928 04:52:21.985709 21210 leader_election.cc:302] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8a66861a15f447b8e9edf8691b9ef9e; no voters:
I20260928 04:52:21.985893 21210 leader_election.cc:288] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:21.985939 21220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:21.986099 21220 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [term 1 LEADER]: Becoming Leader. State: Replica: d8a66861a15f447b8e9edf8691b9ef9e, State: Running, Role: LEADER
I20260928 04:52:21.986137 21210 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:21.986245 21220 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a66861a15f447b8e9edf8691b9ef9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260928 04:52:21.986604 21222 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8a66861a15f447b8e9edf8691b9ef9e. Latest consensus state: current_term: 1 leader_uuid: "d8a66861a15f447b8e9edf8691b9ef9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a66861a15f447b8e9edf8691b9ef9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } } }
I20260928 04:52:21.986730 21222 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:21.986596 21221 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8a66861a15f447b8e9edf8691b9ef9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a66861a15f447b8e9edf8691b9ef9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } } }
I20260928 04:52:21.986867 21221 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:21.987207 21229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:21.987726 21229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:21.989205 21229 catalog_manager.cc:1347] Generated new cluster ID: 334c66d178144b58b57cdfec5fce8c33
I20260928 04:52:21.989251 21229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:22.002698 21229 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:22.002873 21229 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:22.005244 21229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8a66861a15f447b8e9edf8691b9ef9e: Generated new TSK 0
I20260928 04:52:22.005419 21229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:22.008991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 21142
2026-09-28T04:52:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:22 dist-test-slave-jccs krb5kdc[21241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:22 dist-test-slave-jccs krb5kdc[21241](info): set up 2 sockets
Sep 28 04:52:22 dist-test-slave-jccs krb5kdc[21241](info): commencing operation
krb5kdc: starting...
W20260928 04:52:23.021278 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:23 dist-test-slave-jccs krb5kdc[21241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:23.134267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:23.210273 21254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:23.210513 21254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:23.210551 21254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:23.211586 21254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:23.211650 21254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:23.211673 21254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:23.211691 21254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:23.211717 21254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:23.213023 21254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:23.213300 21254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:23.213510 21254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:23.217972 21259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:23.217985 21260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:23.218201 21254 server_base.cc:1034] running on GCE node
W20260928 04:52:23.218005 21262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:23.218472 21254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:23.218747 21254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:23.219907 21254 hybrid_clock.cc:648] HybridClock initialized: now 1790571143219896 us; error 28 us; skew 500 ppm
Sep 28 04:52:23 dist-test-slave-jccs krb5kdc[21241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:23.222283 21254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:23.222719 21254 webserver.cc:460] Webserver started at http://127.0.0.1:35687/ using document root <none> and password file <none>
I20260928 04:52:23.222949 21254 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:23.223012 21254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:23.223187 21254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:23.224180 21254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "649f514b93984c35b73210dffee2a857"
format_stamp: "Formatted at 2026-09-28 04:52:23 on dist-test-slave-jccs"
server_key: "d771fa2ed25701ad2e6ad09f505a914d"
server_key_iv: "407a930ff536068db258976b4f54acff"
server_key_version: "encryptionkey@0"
I20260928 04:52:23.224520 21254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "649f514b93984c35b73210dffee2a857"
format_stamp: "Formatted at 2026-09-28 04:52:23 on dist-test-slave-jccs"
server_key: "d771fa2ed25701ad2e6ad09f505a914d"
server_key_iv: "407a930ff536068db258976b4f54acff"
server_key_version: "encryptionkey@0"
I20260928 04:52:23.225871 21254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 04:52:23.226575 21269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:23.226754 21254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:23.226850 21254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "649f514b93984c35b73210dffee2a857"
format_stamp: "Formatted at 2026-09-28 04:52:23 on dist-test-slave-jccs"
server_key: "d771fa2ed25701ad2e6ad09f505a914d"
server_key_iv: "407a930ff536068db258976b4f54acff"
server_key_version: "encryptionkey@0"
I20260928 04:52:23.226925 21254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:23.235466 21254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:23.237068 21254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:23.237278 21254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:23.240988 21254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38553
I20260928 04:52:23.241016 21321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38553 every 8 connection(s)
I20260928 04:52:23.241386 21254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:23.242170 21322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:23.244375 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857: Bootstrap starting.
I20260928 04:52:23.244954 21322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:23.245311 21322 log.cc:824] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:23.245905 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857: No bootstrap required, opened a new log
I20260928 04:52:23.247331 21322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649f514b93984c35b73210dffee2a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38553 } }
I20260928 04:52:23.247463 21322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:23.247484 21322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649f514b93984c35b73210dffee2a857, State: Initialized, Role: FOLLOWER
I20260928 04:52:23.247656 21322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649f514b93984c35b73210dffee2a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38553 } }
I20260928 04:52:23.247716 21322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:23.247756 21322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:23.247807 21322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:23.248330 21322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649f514b93984c35b73210dffee2a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38553 } }
I20260928 04:52:23.248445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 21254
I20260928 04:52:23.248456 21322 leader_election.cc:302] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 649f514b93984c35b73210dffee2a857; no voters:
I20260928 04:52:23.248567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:23.248613 21322 leader_election.cc:288] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:23.248700 21325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:23.248785 392 external_mini_cluster.cc:1428] Setting key fd5bd004f87d2b870440fab57a70bb67
I20260928 04:52:23.248855 21322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:23.248901 21325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [term 1 LEADER]: Becoming Leader. State: Replica: 649f514b93984c35b73210dffee2a857, State: Running, Role: LEADER
I20260928 04:52:23.248972 21325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649f514b93984c35b73210dffee2a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38553 } }
I20260928 04:52:23.249348 21326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "649f514b93984c35b73210dffee2a857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649f514b93984c35b73210dffee2a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38553 } } }
I20260928 04:52:23.249428 21326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:23.249621 21330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:23.249886 21327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 649f514b93984c35b73210dffee2a857. Latest consensus state: current_term: 1 leader_uuid: "649f514b93984c35b73210dffee2a857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649f514b93984c35b73210dffee2a857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38553 } } }
I20260928 04:52:23.250044 21327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:23.250175 21330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:23.251765 21330 catalog_manager.cc:1347] Generated new cluster ID: 15e3bb050d354ea98137a366f17bcd97
I20260928 04:52:23.251821 21330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:52:23 dist-test-slave-jccs krb5kdc[21241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:52:23.260980 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:23.249723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:34837 (local address 127.0.0.1:38553)
0928 04:52:23.250189 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:23.250199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:23.250230 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:52:23.250341 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:23.250348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:23.250402 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:23.251279 (+ 877us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:23.251382 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:23.252690 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:23.253733 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:23.254369 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:23.254481 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:23.257321 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:23.257324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:23.257325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:23.257338 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:23.259277 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:23.259752 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:23.259755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:23.259755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:23.259855 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:23.260097 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:23.260098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:23.260098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:23.260379 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:23.260526 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:23.260665 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 04:52:23.260758 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":90,"threads_started":1}
I20260928 04:52:23.262072 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:23.269574 21330 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:23.270251 21330 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:23.270740 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:23.263445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:44241 (local address 127.0.0.1:38553)
0928 04:52:23.263481 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:23.263483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:23.263620 (+ 137us) server_negotiation.cc:407] Connection header received
0928 04:52:23.263667 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:23.263668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:23.263677 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:23.263733 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:23.264306 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:23.265103 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:23.266062 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:23.266397 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:23.266470 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:23.267784 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:23.267786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:23.267786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:23.267793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:23.269413 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:23.269821 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:23.269822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:23.269822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:23.269879 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:23.270115 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:23.270116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:23.270116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:23.270250 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:23.270362 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:23.270590 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 04:52:23.270653 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:52:23.275738 21330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 649f514b93984c35b73210dffee2a857: Generated new TSK 0
I20260928 04:52:23.275933 21330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:23.278697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 21254
W20260928 04:52:23.283039 21239 connection.cc:535] client connection to 127.0.0.1:38553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38553): Connection reset by peer (error 104)
2026-09-28T04:52:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:23 dist-test-slave-jccs krb5kdc[21353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:23 dist-test-slave-jccs krb5kdc[21353](info): set up 2 sockets
Sep 28 04:52:23 dist-test-slave-jccs krb5kdc[21353](info): commencing operation
krb5kdc: starting...
W20260928 04:52:24.281143 392 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:24 dist-test-slave-jccs krb5kdc[21353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:24.389778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:24.465703 21366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:24.465896 21366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:24.465919 21366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:24.467067 21366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:24.467106 21366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:24.467121 21366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:24.467133 21366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:24.467145 21366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:24.468567 21366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:24.468807 21366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:24.469026 21366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:24.473584 21374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:24.473762 21372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:24.473714 21366 server_base.cc:1034] running on GCE node
W20260928 04:52:24.473863 21371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:24.474092 21366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:24.474399 21366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:24.475558 21366 hybrid_clock.cc:648] HybridClock initialized: now 1790571144475542 us; error 30 us; skew 500 ppm
Sep 28 04:52:24 dist-test-slave-jccs krb5kdc[21353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:24.483434 21366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:24.484068 21366 webserver.cc:460] Webserver started at http://127.0.0.1:40955/ using document root <none> and password file <none>
I20260928 04:52:24.484293 21366 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:24.484362 21366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:24.484500 21366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:24.485594 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "b8aedfafe6db46d6bc3611841af3d57d"
format_stamp: "Formatted at 2026-09-28 04:52:24 on dist-test-slave-jccs"
server_key: "48fe0cc1ee37f17284d8294d08ff097d"
server_key_iv: "25a2133116ec9bb1ba167deaa500094e"
server_key_version: "encryptionkey@0"
I20260928 04:52:24.485944 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8aedfafe6db46d6bc3611841af3d57d"
format_stamp: "Formatted at 2026-09-28 04:52:24 on dist-test-slave-jccs"
server_key: "48fe0cc1ee37f17284d8294d08ff097d"
server_key_iv: "25a2133116ec9bb1ba167deaa500094e"
server_key_version: "encryptionkey@0"
I20260928 04:52:24.487197 21366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:52:24.487856 21381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:24.488023 21366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:24.488116 21366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "b8aedfafe6db46d6bc3611841af3d57d"
format_stamp: "Formatted at 2026-09-28 04:52:24 on dist-test-slave-jccs"
server_key: "48fe0cc1ee37f17284d8294d08ff097d"
server_key_iv: "25a2133116ec9bb1ba167deaa500094e"
server_key_version: "encryptionkey@0"
I20260928 04:52:24.488196 21366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:24.498837 21366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:24.500924 21366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:24.501075 21366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:24.504916 21366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38569
I20260928 04:52:24.504958 21433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38569 every 8 connection(s)
I20260928 04:52:24.505357 21366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:24.506137 21434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:24.508231 21434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d: Bootstrap starting.
I20260928 04:52:24.508780 21434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:24.509028 21434 log.cc:824] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:24.510842 21434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d: No bootstrap required, opened a new log
I20260928 04:52:24.512351 21434 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8aedfafe6db46d6bc3611841af3d57d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260928 04:52:24.512502 21434 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:24.512542 21434 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8aedfafe6db46d6bc3611841af3d57d, State: Initialized, Role: FOLLOWER
I20260928 04:52:24.512702 21434 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8aedfafe6db46d6bc3611841af3d57d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260928 04:52:24.512782 21434 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:24.512823 21434 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:24.512872 21434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:24.513458 21434 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8aedfafe6db46d6bc3611841af3d57d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260928 04:52:24.513566 21434 leader_election.cc:302] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8aedfafe6db46d6bc3611841af3d57d; no voters:
I20260928 04:52:24.513742 21434 leader_election.cc:288] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:24.513779 21437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:24.513952 21437 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [term 1 LEADER]: Becoming Leader. State: Replica: b8aedfafe6db46d6bc3611841af3d57d, State: Running, Role: LEADER
I20260928 04:52:24.513990 21434 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:24.514268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 21366
I20260928 04:52:24.514389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:24.514605 392 external_mini_cluster.cc:1428] Setting key 62d426ebc41ddb58aef2036722d52357
I20260928 04:52:24.514104 21437 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8aedfafe6db46d6bc3611841af3d57d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260928 04:52:24.515316 21439 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8aedfafe6db46d6bc3611841af3d57d. Latest consensus state: current_term: 1 leader_uuid: "b8aedfafe6db46d6bc3611841af3d57d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8aedfafe6db46d6bc3611841af3d57d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } } }
I20260928 04:52:24.515343 21438 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8aedfafe6db46d6bc3611841af3d57d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8aedfafe6db46d6bc3611841af3d57d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } } }
I20260928 04:52:24.515550 21439 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:24.515558 21438 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d [sys.catalog]: This master's current role is: LEADER
W20260928 04:52:24.515987 21452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:24.516119 21452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:24.516211 21454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:24.516360 21454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:24.518014 21454 catalog_manager.cc:1347] Generated new cluster ID: d1eead65241a4f30a838ff3cdc58814b
I20260928 04:52:24.518065 21454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:24.520706 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:24.515196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:41323 (local address 127.0.0.1:38569)
0928 04:52:24.515371 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:24.515379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:24.515413 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:52:24.515622 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:24.515629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:24.515692 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:24.515825 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:24.516145 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:24.517577 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:24.518665 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:24.519224 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:24.519320 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:24.520069 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:24.520073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:24.520073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:24.520259 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:24.520293 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:24.520306 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:24.520378 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:24.520490 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 04:52:24.520521 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260928 04:52:24.521682 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:24.532542 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:24.525446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:49305 (local address 127.0.0.1:38569)
0928 04:52:24.525479 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:24.525481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:24.529338 (+ 3857us) server_negotiation.cc:407] Connection header received
0928 04:52:24.529368 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:24.529368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:24.529380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:24.529448 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:24.529553 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:24.530284 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:24.531205 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:24.531489 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:24.531587 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:24.532275 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:24.532276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:24.532277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:24.532281 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:24.532295 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:24.532300 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:24.532362 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:24.532438 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 04:52:24.532451 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:52:24.533385 21454 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:24.534031 21454 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:24.544888 21454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8aedfafe6db46d6bc3611841af3d57d: Generated new TSK 0
I20260928 04:52:24.545070 21454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:24.553496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 21366
W20260928 04:52:24.557683 21351 connection.cc:535] client connection to 127.0.0.1:38569 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38569): Connection reset by peer (error 104)
2026-09-28T04:52:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:24 dist-test-slave-jccs krb5kdc[21465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:24 dist-test-slave-jccs krb5kdc[21465](info): set up 2 sockets
Sep 28 04:52:24 dist-test-slave-jccs krb5kdc[21465](info): commencing operation
krb5kdc: starting...
W20260928 04:52:25.545228 392 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:25 dist-test-slave-jccs krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:25.653769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:25.730728 21478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:25.730916 21478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:25.730939 21478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:25.732012 21478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:25.732048 21478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:25.732062 21478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:25.732074 21478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:25.732086 21478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:25.733484 21478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:25.733665 21478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:25.733814 21478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:25.738252 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:25.738310 21486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:25.738274 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:25.738432 21478 server_base.cc:1034] running on GCE node
I20260928 04:52:25.738654 21478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:25.738974 21478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:25.740149 21478 hybrid_clock.cc:648] HybridClock initialized: now 1790571145740129 us; error 34 us; skew 500 ppm
Sep 28 04:52:25 dist-test-slave-jccs krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:25.742486 21478 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:25.742832 21478 webserver.cc:460] Webserver started at http://127.0.0.1:34223/ using document root <none> and password file <none>
I20260928 04:52:25.743024 21478 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:25.743063 21478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:25.743144 21478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:25.744091 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "800d628cee7d4f72b93095a418f3c2ee"
format_stamp: "Formatted at 2026-09-28 04:52:25 on dist-test-slave-jccs"
server_key: "dc6296ec52f5897fdbfa907b399c88e7"
server_key_iv: "77a3055d64abecc95bc7ee2a95002bf3"
server_key_version: "encryptionkey@0"
I20260928 04:52:25.744364 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "800d628cee7d4f72b93095a418f3c2ee"
format_stamp: "Formatted at 2026-09-28 04:52:25 on dist-test-slave-jccs"
server_key: "dc6296ec52f5897fdbfa907b399c88e7"
server_key_iv: "77a3055d64abecc95bc7ee2a95002bf3"
server_key_version: "encryptionkey@0"
I20260928 04:52:25.745673 21478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:52:25.746362 21493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:25.746642 21478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:52:25.746713 21478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "800d628cee7d4f72b93095a418f3c2ee"
format_stamp: "Formatted at 2026-09-28 04:52:25 on dist-test-slave-jccs"
server_key: "dc6296ec52f5897fdbfa907b399c88e7"
server_key_iv: "77a3055d64abecc95bc7ee2a95002bf3"
server_key_version: "encryptionkey@0"
I20260928 04:52:25.746763 21478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:25.749718 21478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:25.751353 21478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:25.751447 21478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:25.755010 21478 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41895
I20260928 04:52:25.755038 21545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41895 every 8 connection(s)
I20260928 04:52:25.755362 21478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:25.756292 21546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:25.757828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 21478
I20260928 04:52:25.757973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:25.758153 392 external_mini_cluster.cc:1428] Setting key f648bcc678dfa355f1d0ba5113b6a2cd
I20260928 04:52:25.758684 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee: Bootstrap starting.
I20260928 04:52:25.759230 21546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:25.759495 21546 log.cc:824] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:25.760272 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:25.758627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:33547 (local address 127.0.0.1:41895)
0928 04:52:25.758786 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:25.758794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:25.758824 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:52:25.759209 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:25.759214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:25.759235 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:25.759405 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:25.759921 (+ 516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:25.759925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:25.759925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:25.759929 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:25.759968 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:25.759976 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:25.760064 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:25.760096 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 04:52:25.760133 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260928 04:52:25.760411 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee: No bootstrap required, opened a new log
I20260928 04:52:25.761059 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:25.762529 21546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d628cee7d4f72b93095a418f3c2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41895 } }
I20260928 04:52:25.762709 21546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:25.762758 21546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800d628cee7d4f72b93095a418f3c2ee, State: Initialized, Role: FOLLOWER
I20260928 04:52:25.762895 21546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d628cee7d4f72b93095a418f3c2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41895 } }
I20260928 04:52:25.763001 21546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:25.763054 21546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:25.763108 21546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:25.763398 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:25.762260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:59639 (local address 127.0.0.1:41895)
0928 04:52:25.762318 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:25.762319 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:25.762488 (+ 169us) server_negotiation.cc:407] Connection header received
0928 04:52:25.762526 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:25.762527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:25.762531 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:25.762580 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:25.763189 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:25.763190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:25.763191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:25.763194 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:25.763208 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:25.763212 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:25.763269 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:25.763348 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 04:52:25.763358 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 04:52:25.763849 21546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d628cee7d4f72b93095a418f3c2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41895 } }
I20260928 04:52:25.763942 21546 leader_election.cc:302] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 800d628cee7d4f72b93095a418f3c2ee; no voters:
I20260928 04:52:25.764122 21546 leader_election.cc:288] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:25.764165 21556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:25.764288 21556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [term 1 LEADER]: Becoming Leader. State: Replica: 800d628cee7d4f72b93095a418f3c2ee, State: Running, Role: LEADER
I20260928 04:52:25.764355 21546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:25.764421 21556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d628cee7d4f72b93095a418f3c2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41895 } }
I20260928 04:52:25.764804 21557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "800d628cee7d4f72b93095a418f3c2ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d628cee7d4f72b93095a418f3c2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41895 } } }
I20260928 04:52:25.764816 21558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800d628cee7d4f72b93095a418f3c2ee. Latest consensus state: current_term: 1 leader_uuid: "800d628cee7d4f72b93095a418f3c2ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800d628cee7d4f72b93095a418f3c2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41895 } } }
I20260928 04:52:25.764922 21557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:25.764930 21558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:25.765105 21565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:25.765681 21565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:25.767148 21565 catalog_manager.cc:1347] Generated new cluster ID: c0d0d9f349324a0fa607cb5996cb51fa
I20260928 04:52:25.767200 21565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:25.775609 21565 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:25.775748 21565 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:25.785231 21565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 800d628cee7d4f72b93095a418f3c2ee: Generated new TSK 0
I20260928 04:52:25.785388 21565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:25.793382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 21478
2026-09-28T04:52:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:25 dist-test-slave-jccs krb5kdc[21577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:25 dist-test-slave-jccs krb5kdc[21577](info): set up 2 sockets
Sep 28 04:52:25 dist-test-slave-jccs krb5kdc[21577](info): commencing operation
krb5kdc: starting...
W20260928 04:52:26.769153 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:26 dist-test-slave-jccs krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:26.878074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:26.956071 21590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:26.956259 21590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:26.956280 21590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:26.957456 21590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:26.957504 21590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:26.957517 21590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:26.957559 21590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:26.957576 21590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:26.958956 21590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34413
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:26.959184 21590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:26.959394 21590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:26.964010 21596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:26.964115 21595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:26.964135 21590 server_base.cc:1034] running on GCE node
W20260928 04:52:26.964022 21598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:26.964385 21590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:26.964664 21590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:26.965806 21590 hybrid_clock.cc:648] HybridClock initialized: now 1790571146965787 us; error 38 us; skew 500 ppm
Sep 28 04:52:26 dist-test-slave-jccs krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:26.975448 21590 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:26.975925 21590 webserver.cc:460] Webserver started at http://127.0.0.1:41291/ using document root <none> and password file <none>
I20260928 04:52:26.976146 21590 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:26.976191 21590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:26.976354 21590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:26.977473 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "7f4e5e1e5366405e81db89b5d802a1cf"
format_stamp: "Formatted at 2026-09-28 04:52:26 on dist-test-slave-jccs"
server_key: "009fe1d2ebbca7a5f0f21d7b7519fda9"
server_key_iv: "930129d351ae8be894d9f282d69b2359"
server_key_version: "encryptionkey@0"
I20260928 04:52:26.977820 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f4e5e1e5366405e81db89b5d802a1cf"
format_stamp: "Formatted at 2026-09-28 04:52:26 on dist-test-slave-jccs"
server_key: "009fe1d2ebbca7a5f0f21d7b7519fda9"
server_key_iv: "930129d351ae8be894d9f282d69b2359"
server_key_version: "encryptionkey@0"
I20260928 04:52:26.979055 21590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:52:26.979719 21605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:26.979878 21590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:26.979978 21590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "7f4e5e1e5366405e81db89b5d802a1cf"
format_stamp: "Formatted at 2026-09-28 04:52:26 on dist-test-slave-jccs"
server_key: "009fe1d2ebbca7a5f0f21d7b7519fda9"
server_key_iv: "930129d351ae8be894d9f282d69b2359"
server_key_version: "encryptionkey@0"
I20260928 04:52:26.980050 21590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:27.001771 21590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:27.003834 21590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:27.003995 21590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:27.007993 21590 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34413
I20260928 04:52:27.008030 21657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34413 every 8 connection(s)
I20260928 04:52:27.008387 21590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:27.009311 21658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:27.011477 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf: Bootstrap starting.
I20260928 04:52:27.011967 21658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:27.012184 21658 log.cc:824] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:27.012529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 21590
I20260928 04:52:27.012656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:27.012835 392 external_mini_cluster.cc:1428] Setting key 2ab5cbf8c1968d8fdad837515f33d783
I20260928 04:52:27.012902 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf: No bootstrap required, opened a new log
I20260928 04:52:27.015182 21658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4e5e1e5366405e81db89b5d802a1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } }
I20260928 04:52:27.015307 21658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:27.015327 21658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4e5e1e5366405e81db89b5d802a1cf, State: Initialized, Role: FOLLOWER
I20260928 04:52:27.015451 21658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4e5e1e5366405e81db89b5d802a1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } }
I20260928 04:52:27.015522 21658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:27.015568 21658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:27.015617 21658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:27.016152 21658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4e5e1e5366405e81db89b5d802a1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } }
I20260928 04:52:27.016233 21658 leader_election.cc:302] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f4e5e1e5366405e81db89b5d802a1cf; no voters:
I20260928 04:52:27.016417 21658 leader_election.cc:288] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:27.016443 21663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:27.016559 21663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [term 1 LEADER]: Becoming Leader. State: Replica: 7f4e5e1e5366405e81db89b5d802a1cf, State: Running, Role: LEADER
I20260928 04:52:27.016685 21658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:27.016697 21663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4e5e1e5366405e81db89b5d802a1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } }
I20260928 04:52:27.017165 21665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f4e5e1e5366405e81db89b5d802a1cf. Latest consensus state: current_term: 1 leader_uuid: "7f4e5e1e5366405e81db89b5d802a1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4e5e1e5366405e81db89b5d802a1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } } }
I20260928 04:52:27.017277 21665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:27.017503 21664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f4e5e1e5366405e81db89b5d802a1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4e5e1e5366405e81db89b5d802a1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34413 } } }
I20260928 04:52:27.017735 21664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:27.017946 21670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:27.018450 21670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:27.020223 21670 catalog_manager.cc:1347] Generated new cluster ID: 14577babf3a84caabc4a848f3efd4018
I20260928 04:52:27.020277 21670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:52:27 dist-test-slave-jccs krb5kdc[21577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 04:52:27.024298 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:27.013312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:48153 (local address 127.0.0.1:34413)
0928 04:52:27.013482 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:27.013490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:27.013528 (+ 38us) server_negotiation.cc:407] Connection header received
0928 04:52:27.014182 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:27.014188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:27.014234 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:27.014342 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:27.015111 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:27.016479 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:27.017503 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:27.018107 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:27.018305 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:27.020875 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:27.020879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:27.020880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:27.020890 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:27.022603 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:27.022997 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:27.023000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:27.023000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:27.023086 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:27.023312 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:27.023313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:27.023313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:27.023554 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:27.023735 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:27.024091 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 04:52:27.024165 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260928 04:52:27.025244 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:27.038187 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:27.028963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:33255 (local address 127.0.0.1:34413)
0928 04:52:27.029321 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:27.029323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:27.029330 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:52:27.029349 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:27.029350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:27.029359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:27.029401 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:27.031071 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:27.032075 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:27.033476 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:27.033954 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:27.034028 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:27.035465 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:27.035467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:27.035467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:27.035474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:27.036871 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:27.037362 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:27.037363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:27.037364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:27.037420 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:27.037650 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:27.037651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:27.037651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:27.037765 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:27.037874 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:27.038008 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 04:52:27.038070 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":336}
I20260928 04:52:27.041188 21670 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:27.041676 21670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:27.052918 21670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f4e5e1e5366405e81db89b5d802a1cf: Generated new TSK 0
I20260928 04:52:27.053136 21670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:27.057281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 21590
W20260928 04:52:27.061472 21575 connection.cc:535] client connection to 127.0.0.1:34413 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34413): Connection reset by peer (error 104)
2026-09-28T04:52:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:27 dist-test-slave-jccs krb5kdc[21689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:27 dist-test-slave-jccs krb5kdc[21689](info): set up 2 sockets
Sep 28 04:52:27 dist-test-slave-jccs krb5kdc[21689](info): commencing operation
krb5kdc: starting...
W20260928 04:52:28.069063 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:28 dist-test-slave-jccs krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:28.175802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:28.253523 21702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:28.253698 21702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:28.253718 21702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:28.254710 21702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:28.254742 21702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:28.254756 21702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:28.254768 21702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:28.254779 21702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:28.256135 21702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34075
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:28.256309 21702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:28.256461 21702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:28.260917 21707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:28.260910 21708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:28.260908 21710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:28.261009 21702 server_base.cc:1034] running on GCE node
I20260928 04:52:28.261353 21702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:28.261610 21702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:28.262763 21702 hybrid_clock.cc:648] HybridClock initialized: now 1790571148262759 us; error 37 us; skew 500 ppm
Sep 28 04:52:28 dist-test-slave-jccs krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:28.265096 21702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:28.265520 21702 webserver.cc:460] Webserver started at http://127.0.0.1:39883/ using document root <none> and password file <none>
I20260928 04:52:28.265699 21702 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:28.265738 21702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:28.265820 21702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:28.266782 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "a9e8c08d207d434197aaec318606efff"
format_stamp: "Formatted at 2026-09-28 04:52:28 on dist-test-slave-jccs"
server_key: "7c5c884e8fc39882aa82618fd15c8a07"
server_key_iv: "b54a568c5fe9467c61cd688d16e99994"
server_key_version: "encryptionkey@0"
I20260928 04:52:28.267066 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9e8c08d207d434197aaec318606efff"
format_stamp: "Formatted at 2026-09-28 04:52:28 on dist-test-slave-jccs"
server_key: "7c5c884e8fc39882aa82618fd15c8a07"
server_key_iv: "b54a568c5fe9467c61cd688d16e99994"
server_key_version: "encryptionkey@0"
I20260928 04:52:28.268200 21702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 04:52:28.268816 21717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:28.268961 21702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:52:28.269040 21702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "a9e8c08d207d434197aaec318606efff"
format_stamp: "Formatted at 2026-09-28 04:52:28 on dist-test-slave-jccs"
server_key: "7c5c884e8fc39882aa82618fd15c8a07"
server_key_iv: "b54a568c5fe9467c61cd688d16e99994"
server_key_version: "encryptionkey@0"
I20260928 04:52:28.269091 21702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:28.281076 21702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:28.282855 21702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:28.282958 21702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:28.286870 21702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34075
I20260928 04:52:28.286926 21769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34075 every 8 connection(s)
I20260928 04:52:28.287230 21702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:28.288146 21770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:28.289954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 21702
I20260928 04:52:28.290046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:28.290212 392 external_mini_cluster.cc:1428] Setting key 5676a264a5e9b2a880a84ba5fb76a02d
I20260928 04:52:28.290328 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff: Bootstrap starting.
I20260928 04:52:28.290984 21770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:28.291281 21770 log.cc:824] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:28.291993 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff: No bootstrap required, opened a new log
I20260928 04:52:28.294094 21770 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e8c08d207d434197aaec318606efff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34075 } }
I20260928 04:52:28.294247 21770 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:28.294279 21770 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e8c08d207d434197aaec318606efff, State: Initialized, Role: FOLLOWER
I20260928 04:52:28.294385 21770 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e8c08d207d434197aaec318606efff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34075 } }
I20260928 04:52:28.294471 21770 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:28.294499 21770 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:28.294544 21770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:28.295022 21770 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e8c08d207d434197aaec318606efff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34075 } }
I20260928 04:52:28.295135 21770 leader_election.cc:302] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e8c08d207d434197aaec318606efff; no voters:
I20260928 04:52:28.295308 21770 leader_election.cc:288] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:28.295331 21775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:28.295464 21775 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [term 1 LEADER]: Becoming Leader. State: Replica: a9e8c08d207d434197aaec318606efff, State: Running, Role: LEADER
I20260928 04:52:28.295591 21770 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:28.295603 21775 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e8c08d207d434197aaec318606efff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34075 } }
I20260928 04:52:28.295958 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:28.290636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:56853 (local address 127.0.0.1:34075)
0928 04:52:28.290792 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:28.290800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:28.290853 (+ 53us) server_negotiation.cc:407] Connection header received
0928 04:52:28.291396 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:28.291405 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:28.291465 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:28.291580 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:28.291808 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:28.293040 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:28.293805 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:28.294392 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:28.294498 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:28.295284 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:28.295288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:28.295288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:28.295535 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:28.295567 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:28.295580 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:28.295668 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:28.295756 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 04:52:28.295782 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260928 04:52:28.296299 21776 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9e8c08d207d434197aaec318606efff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e8c08d207d434197aaec318606efff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34075 } } }
I20260928 04:52:28.296522 21776 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:28.296461 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e8c08d207d434197aaec318606efff. Latest consensus state: current_term: 1 leader_uuid: "a9e8c08d207d434197aaec318606efff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e8c08d207d434197aaec318606efff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34075 } } }
I20260928 04:52:28.296782 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:28.296859 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 04:52:28.297552 21790 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:28.297605 21790 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:28.297676 21786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:28.297837 21786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:28.299237 21786 catalog_manager.cc:1347] Generated new cluster ID: 62651b0321eb48558aa73f158ed30747
I20260928 04:52:28.299292 21786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:28.301880 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:28.298408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:35149 (local address 127.0.0.1:34075)
0928 04:52:28.298439 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:28.298441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:28.298762 (+ 321us) server_negotiation.cc:407] Connection header received
0928 04:52:28.298808 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:28.298827 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:28.298836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:28.298889 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:28.299058 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:28.299744 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:28.300630 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:28.300925 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:28.300998 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:28.301623 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:28.301624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:28.301624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:28.301628 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:28.301641 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:28.301645 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:28.301710 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:28.301793 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 04:52:28.301802 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 04:52:28.317040 21786 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:28.317595 21786 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:28.320318 21786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9e8c08d207d434197aaec318606efff: Generated new TSK 0
I20260928 04:52:28.320493 21786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:28.327497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 21702
W20260928 04:52:28.331286 21687 connection.cc:535] client connection to 127.0.0.1:34075 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34075): Connection reset by peer (error 104)
2026-09-28T04:52:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:28 dist-test-slave-jccs krb5kdc[21801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:28 dist-test-slave-jccs krb5kdc[21801](info): set up 2 sockets
Sep 28 04:52:28 dist-test-slave-jccs krb5kdc[21801](info): commencing operation
krb5kdc: starting...
W20260928 04:52:29.357088 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:29 dist-test-slave-jccs krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 04:52:29.464030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:29.541996 21814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:29.542186 21814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:29.542207 21814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:29.543347 21814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:29.543387 21814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:29.543401 21814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:29.543411 21814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:29.543422 21814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:29.545269 21814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:29.545533 21814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:29.545768 21814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:29.550264 21819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:29.550304 21822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:29.550298 21820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:29.550393 21814 server_base.cc:1034] running on GCE node
I20260928 04:52:29.550783 21814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:29.551091 21814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:29.552285 21814 hybrid_clock.cc:648] HybridClock initialized: now 1790571149552265 us; error 36 us; skew 500 ppm
Sep 28 04:52:29 dist-test-slave-jccs krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:29.554709 21814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 04:52:29.555150 21814 webserver.cc:460] Webserver started at http://127.0.0.1:37903/ using document root <none> and password file <none>
I20260928 04:52:29.555393 21814 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:29.555470 21814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:29.555621 21814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:29.556668 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "86bc2c1d1bf748c78baa9c093157c50f"
format_stamp: "Formatted at 2026-09-28 04:52:29 on dist-test-slave-jccs"
server_key: "9e67891750776c878bcacc51bfffe31f"
server_key_iv: "13b09e8817d02149fa47cce80daea4ff"
server_key_version: "encryptionkey@0"
I20260928 04:52:29.557037 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "86bc2c1d1bf748c78baa9c093157c50f"
format_stamp: "Formatted at 2026-09-28 04:52:29 on dist-test-slave-jccs"
server_key: "9e67891750776c878bcacc51bfffe31f"
server_key_iv: "13b09e8817d02149fa47cce80daea4ff"
server_key_version: "encryptionkey@0"
I20260928 04:52:29.558411 21814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:52:29.559176 21829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:29.559345 21814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:52:29.559433 21814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "86bc2c1d1bf748c78baa9c093157c50f"
format_stamp: "Formatted at 2026-09-28 04:52:29 on dist-test-slave-jccs"
server_key: "9e67891750776c878bcacc51bfffe31f"
server_key_iv: "13b09e8817d02149fa47cce80daea4ff"
server_key_version: "encryptionkey@0"
I20260928 04:52:29.559535 21814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:29.562810 21814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:29.564589 21814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:29.564739 21814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:29.568674 21814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35529
I20260928 04:52:29.568698 21881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35529 every 8 connection(s)
I20260928 04:52:29.568994 21814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:29.569947 21882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:29.572027 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f: Bootstrap starting.
I20260928 04:52:29.572570 21882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:29.572847 21882 log.cc:824] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:29.573529 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f: No bootstrap required, opened a new log
I20260928 04:52:29.575023 21882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc2c1d1bf748c78baa9c093157c50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } }
I20260928 04:52:29.575167 21882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:29.575188 21882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc2c1d1bf748c78baa9c093157c50f, State: Initialized, Role: FOLLOWER
I20260928 04:52:29.575353 21882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc2c1d1bf748c78baa9c093157c50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } }
I20260928 04:52:29.575412 21882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:29.575453 21882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:29.575505 21882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:29.576012 21882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc2c1d1bf748c78baa9c093157c50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } }
I20260928 04:52:29.576117 21882 leader_election.cc:302] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86bc2c1d1bf748c78baa9c093157c50f; no voters:
I20260928 04:52:29.576298 21882 leader_election.cc:288] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:29.576385 21885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:29.576534 21882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:29.576545 21885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [term 1 LEADER]: Becoming Leader. State: Replica: 86bc2c1d1bf748c78baa9c093157c50f, State: Running, Role: LEADER
I20260928 04:52:29.576646 21885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc2c1d1bf748c78baa9c093157c50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } }
I20260928 04:52:29.576982 21887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86bc2c1d1bf748c78baa9c093157c50f. Latest consensus state: current_term: 1 leader_uuid: "86bc2c1d1bf748c78baa9c093157c50f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc2c1d1bf748c78baa9c093157c50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } } }
I20260928 04:52:29.577077 21887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:29.576977 21886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86bc2c1d1bf748c78baa9c093157c50f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86bc2c1d1bf748c78baa9c093157c50f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } } }
I20260928 04:52:29.577122 21886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:29.578280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 21814
I20260928 04:52:29.578402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:29.578605 392 external_mini_cluster.cc:1428] Setting key b44da33d7a5d46ada1e0e67b95d5c935
W20260928 04:52:29.578508 21901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:29.579100 21901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:29.579216 21889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:29.579363 21889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:29.580766 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:29.579042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:58199 (local address 127.0.0.1:35529)
0928 04:52:29.579208 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:29.579219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:29.579349 (+ 130us) server_negotiation.cc:407] Connection header received
0928 04:52:29.579500 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:29.579505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:29.579526 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:29.579649 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:29.580281 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:29.580284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:29.580285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:29.580289 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:29.580324 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:29.580337 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:29.580393 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:29.580513 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 04:52:29.580555 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260928 04:52:29.581189 21889 catalog_manager.cc:1347] Generated new cluster ID: 93b04c3183d740369b46c22b5d2c84f3
I20260928 04:52:29.581233 21889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:29.585762 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:29.590113 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:29.587171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:42863 (local address 127.0.0.1:35529)
0928 04:52:29.587200 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:29.587202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:29.589189 (+ 1987us) server_negotiation.cc:407] Connection header received
0928 04:52:29.589281 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:29.589282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:29.589287 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:29.589341 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:29.589899 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:29.589900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:29.589900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:29.589904 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:29.589918 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:29.589922 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:29.589977 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:29.590028 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 04:52:29.590040 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 04:52:29.597410 21889 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:29.597563 21889 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:29.603073 21889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86bc2c1d1bf748c78baa9c093157c50f: Generated new TSK 0
I20260928 04:52:29.603250 21889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:29.605002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 21814
2026-09-28T04:52:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:29 dist-test-slave-jccs krb5kdc[21913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:29 dist-test-slave-jccs krb5kdc[21913](info): set up 2 sockets
Sep 28 04:52:29 dist-test-slave-jccs krb5kdc[21913](info): commencing operation
krb5kdc: starting...
W20260928 04:52:30.593062 392 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:30 dist-test-slave-jccs krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:30.704107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45547
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:30.782684 21926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:30.782871 21926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:30.782894 21926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:30.784021 21926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:30.784063 21926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:30.784077 21926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:30.784090 21926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:30.784147 21926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:30.785560 21926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:30.785794 21926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:30.786005 21926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:30.790746 21934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:30.790781 21926 server_base.cc:1034] running on GCE node
W20260928 04:52:30.790750 21931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:30.790761 21932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:30.791172 21926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:30.791477 21926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:30.792634 21926 hybrid_clock.cc:648] HybridClock initialized: now 1790571150792630 us; error 37 us; skew 500 ppm
Sep 28 04:52:30 dist-test-slave-jccs krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:30.795001 21926 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:30.795449 21926 webserver.cc:460] Webserver started at http://127.0.98.62:33091/ using document root <none> and password file <none>
I20260928 04:52:30.795660 21926 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:30.795725 21926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:30.795873 21926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:30.796841 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "bb599d0478ec4954b9b0ea95dcc20f85"
format_stamp: "Formatted at 2026-09-28 04:52:30 on dist-test-slave-jccs"
server_key: "1ab02a35d3b588996f6d70dd83ac9b5b"
server_key_iv: "95155c1fc76886c0a8664a08f181957d"
server_key_version: "encryptionkey@0"
I20260928 04:52:30.797266 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb599d0478ec4954b9b0ea95dcc20f85"
format_stamp: "Formatted at 2026-09-28 04:52:30 on dist-test-slave-jccs"
server_key: "1ab02a35d3b588996f6d70dd83ac9b5b"
server_key_iv: "95155c1fc76886c0a8664a08f181957d"
server_key_version: "encryptionkey@0"
I20260928 04:52:30.798573 21926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:52:30.799273 21941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:30.799436 21926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:30.799530 21926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "bb599d0478ec4954b9b0ea95dcc20f85"
format_stamp: "Formatted at 2026-09-28 04:52:30 on dist-test-slave-jccs"
server_key: "1ab02a35d3b588996f6d70dd83ac9b5b"
server_key_iv: "95155c1fc76886c0a8664a08f181957d"
server_key_version: "encryptionkey@0"
I20260928 04:52:30.799614 21926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:30.818305 21926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:30.820549 21926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:30.820710 21926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:30.824723 21926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45547
I20260928 04:52:30.824756 21993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45547 every 8 connection(s)
I20260928 04:52:30.825088 21926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:30.825902 21994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:30.828032 21994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85: Bootstrap starting.
I20260928 04:52:30.828526 21994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:30.828837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 21926
I20260928 04:52:30.828843 21994 log.cc:824] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:30.828950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:30.829236 392 external_mini_cluster.cc:1428] Setting key 309a001ff99fa2b345475af7a986b171
I20260928 04:52:30.829685 21994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85: No bootstrap required, opened a new log
I20260928 04:52:30.831688 21994 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb599d0478ec4954b9b0ea95dcc20f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } }
I20260928 04:52:30.831875 21994 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:30.831918 21994 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb599d0478ec4954b9b0ea95dcc20f85, State: Initialized, Role: FOLLOWER
I20260928 04:52:30.832055 21994 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb599d0478ec4954b9b0ea95dcc20f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } }
I20260928 04:52:30.832155 21994 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:30.832188 21994 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:30.832235 21994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:30.832902 21994 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb599d0478ec4954b9b0ea95dcc20f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } }
I20260928 04:52:30.833009 21994 leader_election.cc:302] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb599d0478ec4954b9b0ea95dcc20f85; no voters:
I20260928 04:52:30.833164 21994 leader_election.cc:288] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:30.833245 21999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:30.833405 21994 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:30.833433 21999 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [term 1 LEADER]: Becoming Leader. State: Replica: bb599d0478ec4954b9b0ea95dcc20f85, State: Running, Role: LEADER
I20260928 04:52:30.833657 21999 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb599d0478ec4954b9b0ea95dcc20f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } }
I20260928 04:52:30.834051 22000 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb599d0478ec4954b9b0ea95dcc20f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb599d0478ec4954b9b0ea95dcc20f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } } }
I20260928 04:52:30.834131 22000 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:30.834597 22001 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb599d0478ec4954b9b0ea95dcc20f85. Latest consensus state: current_term: 1 leader_uuid: "bb599d0478ec4954b9b0ea95dcc20f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb599d0478ec4954b9b0ea95dcc20f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45547 } } }
I20260928 04:52:30.834725 22001 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:30.834842 22008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:30.835392 22008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:30.836903 22008 catalog_manager.cc:1347] Generated new cluster ID: 3175b8451ce04f1b846945f0efcf8e20
I20260928 04:52:30.836956 22008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 04:52:30 dist-test-slave-jccs krb5kdc[21913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:30.841652 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:30.829748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60599 (local address 127.0.98.62:45547)
0928 04:52:30.829927 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:30.829937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:30.829971 (+ 34us) server_negotiation.cc:407] Connection header received
0928 04:52:30.830927 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:30.830942 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:30.831041 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:30.831197 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:30.831783 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:30.833491 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:30.834714 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:30.835309 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:30.835415 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:30.838362 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:30.838366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:30.838366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:30.838378 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:30.840117 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:30.840531 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:30.840534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:30.840534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:30.840610 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:30.840833 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:30.840834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:30.840835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:30.841069 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:30.841288 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:30.841371 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 04:52:30.841468 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260928 04:52:30.842624 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:30.855266 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:30.845526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50213 (local address 127.0.98.62:45547)
0928 04:52:30.845560 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:30.845562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:30.849358 (+ 3796us) server_negotiation.cc:407] Connection header received
0928 04:52:30.849433 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:30.849434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:30.849444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:30.849493 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:30.850016 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:30.850514 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:30.851199 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:30.851414 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:30.851491 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:30.852665 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:30.852667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:30.852667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:30.852674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:30.854110 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:30.854479 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:30.854480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:30.854481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:30.854534 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:30.854768 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:30.854769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:30.854769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:30.854877 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:30.855004 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:30.855105 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 04:52:30.855167 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":17}
I20260928 04:52:30.861961 22008 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:30.862540 22008 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:30.865137 22008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb599d0478ec4954b9b0ea95dcc20f85: Generated new TSK 0
I20260928 04:52:30.865326 22008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:30.872270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 21926
W20260928 04:52:30.876041 21911 connection.cc:535] client connection to 127.0.98.62:45547 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45547): Connection reset by peer (error 104)
2026-09-28T04:52:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1267 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:30 dist-test-slave-jccs krb5kdc[22025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:30 dist-test-slave-jccs krb5kdc[22025](info): set up 2 sockets
Sep 28 04:52:30 dist-test-slave-jccs krb5kdc[22025](info): commencing operation
krb5kdc: starting...
W20260928 04:52:31.873082 392 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:31 dist-test-slave-jccs krb5kdc[22025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:31.981807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43923
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:32.058888 22038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:32.059062 22038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:32.059082 22038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:32.060113 22038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:32.060154 22038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:32.060169 22038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:32.060202 22038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:32.060228 22038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:32.061738 22038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:32.061951 22038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:32.062141 22038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:32.066819 22043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:32.066840 22044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:32.066888 22046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:32.066901 22038 server_base.cc:1034] running on GCE node
I20260928 04:52:32.067203 22038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:32.067512 22038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:32.068673 22038 hybrid_clock.cc:648] HybridClock initialized: now 1790571152068656 us; error 33 us; skew 500 ppm
Sep 28 04:52:32 dist-test-slave-jccs krb5kdc[22025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:32.074321 22038 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:32.074786 22038 webserver.cc:460] Webserver started at http://127.0.98.62:33983/ using document root <none> and password file <none>
I20260928 04:52:32.075013 22038 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:32.075079 22038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:32.075227 22038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:32.076200 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "15078d34dea5445c8e158fbde46b9c97"
format_stamp: "Formatted at 2026-09-28 04:52:32 on dist-test-slave-jccs"
server_key: "30d8bcc4029d267f888d1ab39bcbd28d"
server_key_iv: "e4d2d86761fde3048f1d8a145eaf585e"
server_key_version: "encryptionkey@0"
I20260928 04:52:32.076538 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "15078d34dea5445c8e158fbde46b9c97"
format_stamp: "Formatted at 2026-09-28 04:52:32 on dist-test-slave-jccs"
server_key: "30d8bcc4029d267f888d1ab39bcbd28d"
server_key_iv: "e4d2d86761fde3048f1d8a145eaf585e"
server_key_version: "encryptionkey@0"
I20260928 04:52:32.077875 22038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:52:32.078605 22053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:32.078765 22038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:52:32.078861 22038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "15078d34dea5445c8e158fbde46b9c97"
format_stamp: "Formatted at 2026-09-28 04:52:32 on dist-test-slave-jccs"
server_key: "30d8bcc4029d267f888d1ab39bcbd28d"
server_key_iv: "e4d2d86761fde3048f1d8a145eaf585e"
server_key_version: "encryptionkey@0"
I20260928 04:52:32.078938 22038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:32.089167 22038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:32.090823 22038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:32.090974 22038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:32.094762 22038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43923
I20260928 04:52:32.094790 22105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43923 every 8 connection(s)
I20260928 04:52:32.095171 22038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:32.095638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 22038
I20260928 04:52:32.095772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:32.095976 392 external_mini_cluster.cc:1428] Setting key 1af296ee28b70c55a2a73099b1e1f8a7
I20260928 04:52:32.096167 22106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:32.099109 22106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97: Bootstrap starting.
I20260928 04:52:32.099689 22106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:32.099954 22106 log.cc:824] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:32.100874 22106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97: No bootstrap required, opened a new log
I20260928 04:52:32.101307 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:32.096461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48873 (local address 127.0.98.62:43923)
0928 04:52:32.096642 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:32.096650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:32.096680 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:52:32.097124 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:32.097130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:32.097216 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:32.097334 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:32.097538 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:32.098341 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:32.099369 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:32.099757 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:32.099860 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:32.100611 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:32.100615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:32.100615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:32.100800 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:32.100838 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:32.100852 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:32.100942 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:32.101049 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 04:52:32.101074 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260928 04:52:32.102325 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:32.103336 22106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15078d34dea5445c8e158fbde46b9c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43923 } }
I20260928 04:52:32.103514 22106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:32.103564 22106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15078d34dea5445c8e158fbde46b9c97, State: Initialized, Role: FOLLOWER
I20260928 04:52:32.103690 22106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15078d34dea5445c8e158fbde46b9c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43923 } }
I20260928 04:52:32.103776 22106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:32.103881 22106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:32.103933 22106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:32.104430 22106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15078d34dea5445c8e158fbde46b9c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43923 } }
I20260928 04:52:32.104544 22106 leader_election.cc:302] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15078d34dea5445c8e158fbde46b9c97; no voters:
I20260928 04:52:32.104707 22106 leader_election.cc:288] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:32.104736 22116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:32.104840 22116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [term 1 LEADER]: Becoming Leader. State: Replica: 15078d34dea5445c8e158fbde46b9c97, State: Running, Role: LEADER
I20260928 04:52:32.104945 22106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:32.104975 22116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15078d34dea5445c8e158fbde46b9c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43923 } }
I20260928 04:52:32.105340 22116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15078d34dea5445c8e158fbde46b9c97. Latest consensus state: current_term: 1 leader_uuid: "15078d34dea5445c8e158fbde46b9c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15078d34dea5445c8e158fbde46b9c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43923 } } }
I20260928 04:52:32.105347 22117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15078d34dea5445c8e158fbde46b9c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15078d34dea5445c8e158fbde46b9c97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43923 } } }
I20260928 04:52:32.105517 22116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:32.105752 22117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:32.105763 22125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:32.106329 22125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:32.107117 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:32.103819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38179 (local address 127.0.98.62:43923)
0928 04:52:32.103851 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:32.103853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:32.103948 (+ 95us) server_negotiation.cc:407] Connection header received
0928 04:52:32.104019 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:32.104021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:32.104029 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:32.104085 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:32.104222 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:32.104784 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:32.105777 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:32.106101 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:32.106154 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:32.106871 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:32.106873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:32.106873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:32.106877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:32.106890 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:32.106894 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:32.106924 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:32.107044 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 04:52:32.107053 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 04:52:32.108191 22125 catalog_manager.cc:1347] Generated new cluster ID: b006f942533e481eba4691b71ed68418
I20260928 04:52:32.108238 22125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:32.115911 22125 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:32.116576 22125 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:32.119244 22125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15078d34dea5445c8e158fbde46b9c97: Generated new TSK 0
I20260928 04:52:32.119391 22125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:32.123611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 22038
W20260928 04:52:32.127470 22023 connection.cc:535] client connection to 127.0.98.62:43923 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43923): Connection reset by peer (error 104)
2026-09-28T04:52:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1251 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:32 dist-test-slave-jccs krb5kdc[22137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:32 dist-test-slave-jccs krb5kdc[22137](info): set up 2 sockets
Sep 28 04:52:32 dist-test-slave-jccs krb5kdc[22137](info): commencing operation
krb5kdc: starting...
W20260928 04:52:33.141322 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:33 dist-test-slave-jccs krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:33.255373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38925
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:33.334950 22150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:33.335135 22150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:33.335157 22150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:33.336315 22150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:33.336356 22150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:33.336370 22150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:33.336383 22150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:33.336395 22150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:33.337903 22150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:33.338132 22150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:33.338383 22150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:33.343251 22155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:33.343376 22156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:33.343279 22158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:33.343433 22150 server_base.cc:1034] running on GCE node
I20260928 04:52:33.343737 22150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:33.344053 22150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:33.345253 22150 hybrid_clock.cc:648] HybridClock initialized: now 1790571153345225 us; error 26 us; skew 500 ppm
Sep 28 04:52:33 dist-test-slave-jccs krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:33.347863 22150 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:33.348318 22150 webserver.cc:460] Webserver started at http://127.0.98.62:34537/ using document root <none> and password file <none>
I20260928 04:52:33.348575 22150 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:33.348645 22150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:33.348796 22150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:33.350023 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "582a8add53e4417aacdd6ba1d54416b4"
format_stamp: "Formatted at 2026-09-28 04:52:33 on dist-test-slave-jccs"
server_key: "bb6b456cfd74b3a60ba6b72943d1b42a"
server_key_iv: "837896bddeb38351411327d866c8a4c0"
server_key_version: "encryptionkey@0"
I20260928 04:52:33.350385 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "582a8add53e4417aacdd6ba1d54416b4"
format_stamp: "Formatted at 2026-09-28 04:52:33 on dist-test-slave-jccs"
server_key: "bb6b456cfd74b3a60ba6b72943d1b42a"
server_key_iv: "837896bddeb38351411327d866c8a4c0"
server_key_version: "encryptionkey@0"
I20260928 04:52:33.351728 22150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 04:52:33.352417 22165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:33.352582 22150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:33.352685 22150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "582a8add53e4417aacdd6ba1d54416b4"
format_stamp: "Formatted at 2026-09-28 04:52:33 on dist-test-slave-jccs"
server_key: "bb6b456cfd74b3a60ba6b72943d1b42a"
server_key_iv: "837896bddeb38351411327d866c8a4c0"
server_key_version: "encryptionkey@0"
I20260928 04:52:33.352773 22150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:33.356014 22150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:33.357865 22150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:33.358021 22150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:33.361838 22150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38925
I20260928 04:52:33.361876 22217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38925 every 8 connection(s)
I20260928 04:52:33.362182 22150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:33.363125 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:33.365303 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4: Bootstrap starting.
I20260928 04:52:33.365857 22218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:33.366138 22218 log.cc:824] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:33.366786 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4: No bootstrap required, opened a new log
I20260928 04:52:33.368319 22218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582a8add53e4417aacdd6ba1d54416b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } }
I20260928 04:52:33.368450 22218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:33.368472 22218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582a8add53e4417aacdd6ba1d54416b4, State: Initialized, Role: FOLLOWER
I20260928 04:52:33.368672 22218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582a8add53e4417aacdd6ba1d54416b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } }
I20260928 04:52:33.368752 22218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:33.368777 22218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:33.368840 22218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:33.369426 22218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582a8add53e4417aacdd6ba1d54416b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } }
I20260928 04:52:33.369531 22218 leader_election.cc:302] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 582a8add53e4417aacdd6ba1d54416b4; no voters:
I20260928 04:52:33.369699 22218 leader_election.cc:288] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:33.369738 22221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:33.369897 22221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [term 1 LEADER]: Becoming Leader. State: Replica: 582a8add53e4417aacdd6ba1d54416b4, State: Running, Role: LEADER
I20260928 04:52:33.369943 22218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:33.370025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 22150
I20260928 04:52:33.370025 22221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582a8add53e4417aacdd6ba1d54416b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } }
I20260928 04:52:33.370165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:33.370371 392 external_mini_cluster.cc:1428] Setting key 91416f46d75e998c218c9d0369fb9e00
I20260928 04:52:33.370417 22221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 582a8add53e4417aacdd6ba1d54416b4. Latest consensus state: current_term: 1 leader_uuid: "582a8add53e4417aacdd6ba1d54416b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582a8add53e4417aacdd6ba1d54416b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } } }
I20260928 04:52:33.370428 22222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "582a8add53e4417aacdd6ba1d54416b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582a8add53e4417aacdd6ba1d54416b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38925 } } }
I20260928 04:52:33.370530 22222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:33.370489 22221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:33.371136 22231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:33.371685 22231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:33.373239 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:33.370819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37437 (local address 127.0.98.62:38925)
0928 04:52:33.371537 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:33.371548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:33.371581 (+ 33us) server_negotiation.cc:407] Connection header received
0928 04:52:33.371697 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:33.371721 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:33.371750 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:33.372169 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:33.372628 (+ 459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:33.372632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:33.372632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:33.372636 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:33.372682 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:33.372696 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:33.372920 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:33.372994 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 04:52:33.373036 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":86,"threads_started":1}
I20260928 04:52:33.373329 22231 catalog_manager.cc:1347] Generated new cluster ID: 5fc5512dce7e4cb6ad5f2bb2bd812f76
I20260928 04:52:33.373400 22231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:33.374182 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:33.376613 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:33.375542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35215 (local address 127.0.98.62:38925)
0928 04:52:33.375573 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:33.375575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:33.375773 (+ 198us) server_negotiation.cc:407] Connection header received
0928 04:52:33.375813 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:33.375813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:33.375818 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:33.375870 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:33.376421 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:33.376422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:33.376422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:33.376426 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:33.376439 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:33.376443 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:33.376491 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:33.376541 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 04:52:33.376554 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 04:52:33.383607 22231 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:33.383754 22231 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:33.391793 22231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 582a8add53e4417aacdd6ba1d54416b4: Generated new TSK 0
I20260928 04:52:33.391973 22231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:33.396881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 22150
2026-09-28T04:52:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:33 dist-test-slave-jccs krb5kdc[22249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:33 dist-test-slave-jccs krb5kdc[22249](info): set up 2 sockets
Sep 28 04:52:33 dist-test-slave-jccs krb5kdc[22249](info): commencing operation
krb5kdc: starting...
W20260928 04:52:34.409255 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:34 dist-test-slave-jccs krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:34.524166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35671
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:34.602550 22262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:34.602842 22262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:34.602882 22262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:34.603998 22262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:34.604063 22262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:34.604086 22262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:34.604106 22262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:34.604135 22262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:34.605644 22262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:34.605902 22262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:34.606112 22262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:34.610639 22270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:34.610656 22268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:34.610649 22267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:34.610878 22262 server_base.cc:1034] running on GCE node
I20260928 04:52:34.611145 22262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:34.611379 22262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:34.612569 22262 hybrid_clock.cc:648] HybridClock initialized: now 1790571154612560 us; error 27 us; skew 500 ppm
Sep 28 04:52:34 dist-test-slave-jccs krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:34.614930 22262 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:34.615353 22262 webserver.cc:460] Webserver started at http://127.0.98.62:46211/ using document root <none> and password file <none>
I20260928 04:52:34.615592 22262 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:34.615636 22262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:34.615841 22262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:34.616850 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "6e95c47debcb4624906a6808a7bf9750"
format_stamp: "Formatted at 2026-09-28 04:52:34 on dist-test-slave-jccs"
server_key: "57dec1b780078f72e7d5734fd364de34"
server_key_iv: "42636a921ccc31e52d3d22419abc4a80"
server_key_version: "encryptionkey@0"
I20260928 04:52:34.617244 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e95c47debcb4624906a6808a7bf9750"
format_stamp: "Formatted at 2026-09-28 04:52:34 on dist-test-slave-jccs"
server_key: "57dec1b780078f72e7d5734fd364de34"
server_key_iv: "42636a921ccc31e52d3d22419abc4a80"
server_key_version: "encryptionkey@0"
I20260928 04:52:34.618502 22262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:52:34.619166 22277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:34.619331 22262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:52:34.619410 22262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "6e95c47debcb4624906a6808a7bf9750"
format_stamp: "Formatted at 2026-09-28 04:52:34 on dist-test-slave-jccs"
server_key: "57dec1b780078f72e7d5734fd364de34"
server_key_iv: "42636a921ccc31e52d3d22419abc4a80"
server_key_version: "encryptionkey@0"
I20260928 04:52:34.619463 22262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:34.629635 22262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:34.631736 22262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:34.631840 22262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:34.635357 22262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35671
I20260928 04:52:34.635417 22329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35671 every 8 connection(s)
I20260928 04:52:34.635718 22262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:34.636555 22330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:34.638533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 22262
I20260928 04:52:34.638649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:34.638857 392 external_mini_cluster.cc:1428] Setting key 7df4eb9daa2da558cdff5965f94ef41e
I20260928 04:52:34.639113 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750: Bootstrap starting.
I20260928 04:52:34.639781 22330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:34.640074 22330 log.cc:824] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:34.640707 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750: No bootstrap required, opened a new log
I20260928 04:52:34.642931 22330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e95c47debcb4624906a6808a7bf9750" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35671 } }
I20260928 04:52:34.643108 22330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:34.643142 22330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e95c47debcb4624906a6808a7bf9750, State: Initialized, Role: FOLLOWER
I20260928 04:52:34.643249 22330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e95c47debcb4624906a6808a7bf9750" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35671 } }
I20260928 04:52:34.643314 22330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:34.643342 22330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:34.643384 22330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:34.644019 22330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e95c47debcb4624906a6808a7bf9750" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35671 } }
I20260928 04:52:34.644142 22330 leader_election.cc:302] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e95c47debcb4624906a6808a7bf9750; no voters:
I20260928 04:52:34.644328 22330 leader_election.cc:288] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:34.644383 22335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:34.644523 22335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [term 1 LEADER]: Becoming Leader. State: Replica: 6e95c47debcb4624906a6808a7bf9750, State: Running, Role: LEADER
I20260928 04:52:34.644583 22330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:34.644644 22335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e95c47debcb4624906a6808a7bf9750" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35671 } }
I20260928 04:52:34.644989 22335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e95c47debcb4624906a6808a7bf9750. Latest consensus state: current_term: 1 leader_uuid: "6e95c47debcb4624906a6808a7bf9750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e95c47debcb4624906a6808a7bf9750" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35671 } } }
I20260928 04:52:34.645153 22337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e95c47debcb4624906a6808a7bf9750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e95c47debcb4624906a6808a7bf9750" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35671 } } }
I20260928 04:52:34.645359 22337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:34.645206 22335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:34.645848 22344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 04:52:34 dist-test-slave-jccs krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:34.646350 22344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:34.648873 22344 catalog_manager.cc:1347] Generated new cluster ID: eea5e36e58df4ecfaa695e1f643ba7f4
I20260928 04:52:34.648934 22344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:34.650118 22333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:34.639366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34247 (local address 127.0.98.62:35671)
0928 04:52:34.639554 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:34.639563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:34.639600 (+ 37us) server_negotiation.cc:407] Connection header received
0928 04:52:34.640075 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:34.640082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:34.640127 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:34.640276 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:34.640915 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:34.642281 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:34.643009 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:34.643595 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:34.643729 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:34.646651 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:34.646654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:34.646655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:34.646666 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:34.648558 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:34.648977 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:34.648980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:34.648980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:34.649072 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:34.649323 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:34.649330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:34.649331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:34.649558 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:34.649716 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:34.649823 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 04:52:34.649952 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260928 04:52:34.654059 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:34.663115 22333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:34.655358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56151 (local address 127.0.98.62:35671)
0928 04:52:34.655390 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:34.655392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:34.657355 (+ 1963us) server_negotiation.cc:407] Connection header received
0928 04:52:34.657443 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:34.657444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:34.657453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:34.657496 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:34.657999 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:34.658473 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:34.659193 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:34.659412 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:34.659487 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:34.660635 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:34.660636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:34.660636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:34.660643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:34.662055 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:34.662398 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:34.662399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:34.662399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:34.662449 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:34.662642 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:34.662643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:34.662643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:34.662749 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:34.662864 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:34.662956 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 04:52:34.663019 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:52:34.676693 22344 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:34.677471 22344 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:34.683182 22344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e95c47debcb4624906a6808a7bf9750: Generated new TSK 0
I20260928 04:52:34.683378 22344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:34.686581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 22262
2026-09-28T04:52:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:34 dist-test-slave-jccs krb5kdc[22361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:34 dist-test-slave-jccs krb5kdc[22361](info): set up 2 sockets
Sep 28 04:52:34 dist-test-slave-jccs krb5kdc[22361](info): commencing operation
krb5kdc: starting...
W20260928 04:52:35.701328 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:35 dist-test-slave-jccs krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:35.811214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:35.888581 22374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:35.888804 22374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:35.888837 22374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:35.889937 22374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:35.890002 22374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:35.890023 22374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:35.890041 22374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:35.890067 22374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:35.891433 22374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:35.891674 22374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:35.891881 22374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:35.896381 22382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:35.896390 22380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:35.896523 22374 server_base.cc:1034] running on GCE node
W20260928 04:52:35.896397 22379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:35.896832 22374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:35.897125 22374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:35.898294 22374 hybrid_clock.cc:648] HybridClock initialized: now 1790571155898288 us; error 52 us; skew 500 ppm
Sep 28 04:52:35 dist-test-slave-jccs krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:35.900677 22374 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:35.901063 22374 webserver.cc:460] Webserver started at http://127.0.98.62:35755/ using document root <none> and password file <none>
I20260928 04:52:35.901328 22374 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:35.901393 22374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:35.901540 22374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:35.902473 22374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "7d59d987f7354323accfbe402cec6aee"
format_stamp: "Formatted at 2026-09-28 04:52:35 on dist-test-slave-jccs"
server_key: "d61d8ce6809630ba247b8b348ec497e4"
server_key_iv: "f2348012ec781868367d8497de53a28f"
server_key_version: "encryptionkey@0"
I20260928 04:52:35.902810 22374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d59d987f7354323accfbe402cec6aee"
format_stamp: "Formatted at 2026-09-28 04:52:35 on dist-test-slave-jccs"
server_key: "d61d8ce6809630ba247b8b348ec497e4"
server_key_iv: "f2348012ec781868367d8497de53a28f"
server_key_version: "encryptionkey@0"
I20260928 04:52:35.904055 22374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:35.904740 22389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:35.904922 22374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:35.904999 22374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "7d59d987f7354323accfbe402cec6aee"
format_stamp: "Formatted at 2026-09-28 04:52:35 on dist-test-slave-jccs"
server_key: "d61d8ce6809630ba247b8b348ec497e4"
server_key_iv: "f2348012ec781868367d8497de53a28f"
server_key_version: "encryptionkey@0"
I20260928 04:52:35.905052 22374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:35.921339 22374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:35.923087 22374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:35.923189 22374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:35.926970 22374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38493
I20260928 04:52:35.927024 22441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38493 every 8 connection(s)
I20260928 04:52:35.927347 22374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:35.928220 22442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:35.930431 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee: Bootstrap starting.
I20260928 04:52:35.930940 22442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:35.931161 22442 log.cc:824] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:35.931756 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee: No bootstrap required, opened a new log
I20260928 04:52:35.933219 22442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d59d987f7354323accfbe402cec6aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38493 } }
I20260928 04:52:35.933359 22442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:35.933384 22442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d59d987f7354323accfbe402cec6aee, State: Initialized, Role: FOLLOWER
I20260928 04:52:35.933537 22442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d59d987f7354323accfbe402cec6aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38493 } }
I20260928 04:52:35.933610 22442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:35.933632 22442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:35.933660 22442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:35.934105 22442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d59d987f7354323accfbe402cec6aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38493 } }
I20260928 04:52:35.934185 22442 leader_election.cc:302] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d59d987f7354323accfbe402cec6aee; no voters:
I20260928 04:52:35.934299 22442 leader_election.cc:288] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:35.934361 22445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:35.934535 22445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [term 1 LEADER]: Becoming Leader. State: Replica: 7d59d987f7354323accfbe402cec6aee, State: Running, Role: LEADER
I20260928 04:52:35.934574 22442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:35.934618 22445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d59d987f7354323accfbe402cec6aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38493 } }
I20260928 04:52:35.935002 22446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d59d987f7354323accfbe402cec6aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d59d987f7354323accfbe402cec6aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38493 } } }
I20260928 04:52:35.935133 22446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:35.935001 22447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d59d987f7354323accfbe402cec6aee. Latest consensus state: current_term: 1 leader_uuid: "7d59d987f7354323accfbe402cec6aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d59d987f7354323accfbe402cec6aee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38493 } } }
I20260928 04:52:35.935315 22449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:35.935324 22447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:35.935477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 22374
I20260928 04:52:35.935612 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:35.935801 392 external_mini_cluster.cc:1428] Setting key fc37a6ccaabc1a900e51a11ea4eebdce
I20260928 04:52:35.935828 22449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:35.937549 22449 catalog_manager.cc:1347] Generated new cluster ID: 9a38c22d706c449ca9d30ef90af8674e
I20260928 04:52:35.937601 22449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:35.944844 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:35.936226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57391 (local address 127.0.98.62:38493)
0928 04:52:35.936929 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:35.936937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:35.936968 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:52:35.937083 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:35.937089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:35.937143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:35.937304 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:35.941260 (+ 3956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:35.942166 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:35.942946 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:35.943338 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:35.943490 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:35.944179 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:35.944182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:35.944183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:35.944357 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:35.944392 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:35.944402 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:35.944473 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:35.944544 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 04:52:35.944619 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":66,"threads_started":1}
I20260928 04:52:35.945135 22449 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:35.945785 22449 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:35.945798 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:35.949203 22449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d59d987f7354323accfbe402cec6aee: Generated new TSK 0
I20260928 04:52:35.949373 22449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:35.950407 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:35.947130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56249 (local address 127.0.98.62:38493)
0928 04:52:35.947163 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:35.947165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:35.947305 (+ 140us) server_negotiation.cc:407] Connection header received
0928 04:52:35.947358 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:35.947359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:35.947369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:35.947424 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:35.947541 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:35.948227 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:35.949315 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:35.949584 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:35.949638 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:35.950204 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:35.950205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:35.950205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:35.950208 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:35.950219 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:35.950222 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:35.950256 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:35.950314 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 04:52:35.950335 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:52:35.952071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 22374
2026-09-28T04:52:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:36 dist-test-slave-jccs krb5kdc[22473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:36 dist-test-slave-jccs krb5kdc[22473](info): set up 2 sockets
Sep 28 04:52:36 dist-test-slave-jccs krb5kdc[22473](info): commencing operation
krb5kdc: starting...
W20260928 04:52:36.957312 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:37 dist-test-slave-jccs krb5kdc[22473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:37.069634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33535
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:37.146938 22486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:37.147161 22486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:37.147197 22486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:37.148248 22486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:37.148324 22486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:37.148347 22486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:37.148377 22486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:37.148398 22486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:37.149868 22486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:37.150126 22486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:37.150341 22486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:37.155493 22494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:37.155558 22486 server_base.cc:1034] running on GCE node
W20260928 04:52:37.155548 22492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:37.155514 22491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:37.155923 22486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:37.156132 22486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:37.157289 22486 hybrid_clock.cc:648] HybridClock initialized: now 1790571157157274 us; error 36 us; skew 500 ppm
Sep 28 04:52:37 dist-test-slave-jccs krb5kdc[22473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:37.164589 22486 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:37.165019 22486 webserver.cc:460] Webserver started at http://127.0.98.62:44775/ using document root <none> and password file <none>
I20260928 04:52:37.165283 22486 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:37.165350 22486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:37.165499 22486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:37.166461 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "a865cb20fdaf4491b3dab44a8eda05e3"
format_stamp: "Formatted at 2026-09-28 04:52:37 on dist-test-slave-jccs"
server_key: "42124ef29af8dcb1b1600be014700a2a"
server_key_iv: "eccbe33adc103400e4fb03eb11e6a4b4"
server_key_version: "encryptionkey@0"
I20260928 04:52:37.166818 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "a865cb20fdaf4491b3dab44a8eda05e3"
format_stamp: "Formatted at 2026-09-28 04:52:37 on dist-test-slave-jccs"
server_key: "42124ef29af8dcb1b1600be014700a2a"
server_key_iv: "eccbe33adc103400e4fb03eb11e6a4b4"
server_key_version: "encryptionkey@0"
I20260928 04:52:37.168040 22486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:37.168722 22501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:37.168879 22486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:37.168967 22486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "a865cb20fdaf4491b3dab44a8eda05e3"
format_stamp: "Formatted at 2026-09-28 04:52:37 on dist-test-slave-jccs"
server_key: "42124ef29af8dcb1b1600be014700a2a"
server_key_iv: "eccbe33adc103400e4fb03eb11e6a4b4"
server_key_version: "encryptionkey@0"
I20260928 04:52:37.169044 22486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:37.172107 22486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:37.174325 22486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:37.174489 22486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:37.178218 22486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33535
I20260928 04:52:37.178248 22553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33535 every 8 connection(s)
I20260928 04:52:37.178547 22486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:37.179471 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:37.181627 22554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3: Bootstrap starting.
I20260928 04:52:37.182138 22554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:37.182402 22554 log.cc:824] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:37.183007 22554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3: No bootstrap required, opened a new log
I20260928 04:52:37.183969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 22486
I20260928 04:52:37.184080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:37.184280 392 external_mini_cluster.cc:1428] Setting key 683864d8b0d2f69b9b4a21ca3e5a2000
I20260928 04:52:37.184893 22554 raft_consensus.cc:348] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865cb20fdaf4491b3dab44a8eda05e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33535 } }
I20260928 04:52:37.185050 22554 raft_consensus.cc:374] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 04:52:37.185092 22554 raft_consensus.cc:729] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a865cb20fdaf4491b3dab44a8eda05e3, State: Initialized, Role: FOLLOWER
I20260928 04:52:37.185242 22554 consensus_queue.cc:260] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865cb20fdaf4491b3dab44a8eda05e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33535 } }
I20260928 04:52:37.185322 22554 raft_consensus.cc:388] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 04:52:37.185379 22554 raft_consensus.cc:482] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 04:52:37.185431 22554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:37.186272 22554 raft_consensus.cc:504] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865cb20fdaf4491b3dab44a8eda05e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33535 } }
I20260928 04:52:37.186386 22554 leader_election.cc:302] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a865cb20fdaf4491b3dab44a8eda05e3; no voters:
I20260928 04:52:37.186421 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:37.184770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43743 (local address 127.0.98.62:33535)
0928 04:52:37.184944 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:37.184954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:37.184986 (+ 32us) server_negotiation.cc:407] Connection header received
0928 04:52:37.185099 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:37.185106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:37.185141 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:37.185294 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:37.185972 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:37.185976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:37.185976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:37.185981 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:37.186026 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:37.186048 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:37.186152 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:37.186190 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 04:52:37.186229 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260928 04:52:37.186532 22554 leader_election.cc:288] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 04:52:37.186568 22559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:37.186733 22559 raft_consensus.cc:686] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [term 1 LEADER]: Becoming Leader. State: Replica: a865cb20fdaf4491b3dab44a8eda05e3, State: Running, Role: LEADER
I20260928 04:52:37.186771 22554 sys_catalog.cc:564] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:37.186904 22559 consensus_queue.cc:237] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865cb20fdaf4491b3dab44a8eda05e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33535 } }
I20260928 04:52:37.187238 22561 sys_catalog.cc:455] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a865cb20fdaf4491b3dab44a8eda05e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865cb20fdaf4491b3dab44a8eda05e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33535 } } }
I20260928 04:52:37.187335 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:37.187264 22560 sys_catalog.cc:455] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a865cb20fdaf4491b3dab44a8eda05e3. Latest consensus state: current_term: 1 leader_uuid: "a865cb20fdaf4491b3dab44a8eda05e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a865cb20fdaf4491b3dab44a8eda05e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33535 } } }
I20260928 04:52:37.187381 22560 sys_catalog.cc:458] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:37.187507 22561 sys_catalog.cc:458] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:37.188072 22568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:37.188576 22568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:37.190003 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:37.188754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48723 (local address 127.0.98.62:33535)
0928 04:52:37.188804 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:37.188806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:37.189046 (+ 240us) server_negotiation.cc:407] Connection header received
0928 04:52:37.189121 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:37.189122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:37.189127 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:37.189194 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:37.189796 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:37.189797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:37.189798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 04:52:37.189802 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:37.189814 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 04:52:37.189818 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:37.189872 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:37.189925 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 04:52:37.189938 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 04:52:37.190079 22568 catalog_manager.cc:1347] Generated new cluster ID: ecbbce16bc654c32b6d98b38beef1f12
I20260928 04:52:37.190130 22568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:37.209044 22568 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:37.209285 22568 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:37.221757 22568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a865cb20fdaf4491b3dab44a8eda05e3: Generated new TSK 0
I20260928 04:52:37.221952 22568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 04:52:37.229957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 22486
2026-09-28T04:52:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1278 ms)
[----------] 18 tests from AuthTokenIssuingTest (22806 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:37 dist-test-slave-jccs krb5kdc[22585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:37 dist-test-slave-jccs krb5kdc[22585](info): set up 2 sockets
Sep 28 04:52:37 dist-test-slave-jccs krb5kdc[22585](info): commencing operation
krb5kdc: starting...
W20260928 04:52:38.249102 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:38.359912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43655
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43655,127.0.98.61:45161,127.0.98.60:44655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:38.435552 22598 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:52:38.435801 22598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:38.435827 22598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:38.435885 22598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:38.436951 22598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:38.437012 22598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:38.437036 22598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:38.437057 22598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:38.437084 22598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:38.438510 22598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43655,127.0.98.61:45161,127.0.98.60:44655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:38.438813 22598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:38.439030 22598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:38.443610 22603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:38.443703 22604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:38.443769 22598 server_base.cc:1034] running on GCE node
W20260928 04:52:38.443698 22606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:38.444131 22598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:38.444366 22598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:38.445510 22598 hybrid_clock.cc:648] HybridClock initialized: now 1790571158445490 us; error 35 us; skew 500 ppm
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:38.447808 22598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:38.448145 22598 webserver.cc:460] Webserver started at http://127.0.98.62:40721/ using document root <none> and password file <none>
I20260928 04:52:38.448321 22598 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:38.448359 22598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:38.448441 22598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:38.449427 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "8d9c07988d114f9c967190d7d6e95097"
format_stamp: "Formatted at 2026-09-28 04:52:38 on dist-test-slave-jccs"
server_key: "7a6ad0df7ae75b35418f7a75defbeda0"
server_key_iv: "7ead37d70912158666540bb0b337db60"
server_key_version: "encryptionkey@0"
I20260928 04:52:38.449707 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d9c07988d114f9c967190d7d6e95097"
format_stamp: "Formatted at 2026-09-28 04:52:38 on dist-test-slave-jccs"
server_key: "7a6ad0df7ae75b35418f7a75defbeda0"
server_key_iv: "7ead37d70912158666540bb0b337db60"
server_key_version: "encryptionkey@0"
I20260928 04:52:38.450908 22598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 04:52:38.451552 22613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:38.451740 22598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:38.451817 22598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "8d9c07988d114f9c967190d7d6e95097"
format_stamp: "Formatted at 2026-09-28 04:52:38 on dist-test-slave-jccs"
server_key: "7a6ad0df7ae75b35418f7a75defbeda0"
server_key_iv: "7ead37d70912158666540bb0b337db60"
server_key_version: "encryptionkey@0"
I20260928 04:52:38.451869 22598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:38.476264 22598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:38.478619 22598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:38.478750 22598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:38.482482 22598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43655
I20260928 04:52:38.482553 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43655 every 8 connection(s)
I20260928 04:52:38.482898 22598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:38.483867 22666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:38.483906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 22598
I20260928 04:52:38.484011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:38.484215 392 external_mini_cluster.cc:1428] Setting key 5040faf550cd711f6ba5505ff4d1c78a
I20260928 04:52:38.485664 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:38.498085 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.486443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43655 (local address 127.0.0.1:56658)
0928 04:52:38.486605 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.486641 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:38.487079 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:38.487363 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:38.487365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:38.487660 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:38.487812 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.487816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.489422 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.489422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.490089 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.490090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.490794 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.490795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.490848 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.491338 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:38.491346 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:38.494855 (+ 3509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:38.496589 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.496595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.496597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.496790 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.497026 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.497027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.497027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.497115 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.497495 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:38.497500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:38.497557 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 04:52:38.497662 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 04:52:38.497907 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":89,"thread_start_us":94,"threads_started":1}
I20260928 04:52:38.498093 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.486452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.0.98.62:43655)
0928 04:52:38.486619 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.486629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.486718 (+ 89us) server_negotiation.cc:407] Connection header received
0928 04:52:38.487228 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.487231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.487253 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.487366 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.487913 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.489327 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.490175 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.490687 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.490812 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.494956 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.494963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.494964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.494970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.496487 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.496880 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.496883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.496883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.496960 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.497208 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.497208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.497209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.497410 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.497516 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.497685 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.497847 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260928 04:52:38.498946 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:38.499298 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.499191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45161 (local address 127.0.0.1:33884)
0928 04:52:38.499241 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.499280 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 04:52:38.499428 22614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45161: connect: Connection refused (error 111)
I20260928 04:52:38.499832 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.499740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45161 (local address 127.0.0.1:33892)
0928 04:52:38.499787 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.499810 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 04:52:38.499933 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45161: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45161: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 04:52:38.508669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45161
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43655,127.0.98.61:45161,127.0.98.60:44655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:52:38.549217 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } attempt: 1
I20260928 04:52:38.549705 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.549552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45161 (local address 127.0.0.1:33896)
0928 04:52:38.549618 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.549668 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 04:52:38.550122 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.550015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45161 (local address 127.0.0.1:33904)
0928 04:52:38.550060 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.550101 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 04:52:38.550248 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45161: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45161: connect: Connection refused (error 111)
W20260928 04:52:38.586997 22673 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:52:38.587209 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:38.587226 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:38.587246 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:38.588281 22673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:38.588311 22673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:38.588325 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:38.588338 22673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:38.588351 22673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:38.589720 22673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43655,127.0.98.61:45161,127.0.98.60:44655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:38.589893 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:38.590039 22673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:38.594542 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:38.594532 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:38.594533 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:38.594810 22673 server_base.cc:1034] running on GCE node
I20260928 04:52:38.595052 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:38.595323 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:38.596503 22673 hybrid_clock.cc:648] HybridClock initialized: now 1790571158596484 us; error 29 us; skew 500 ppm
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:38.598994 22673 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 04:52:38.599371 22673 webserver.cc:460] Webserver started at http://127.0.98.61:43633/ using document root <none> and password file <none>
I20260928 04:52:38.599541 22673 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:38.599581 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:38.599666 22673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:38.600637 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/data/instance:
uuid: "947a570d3172405e92adaa1c662b271a"
format_stamp: "Formatted at 2026-09-28 04:52:38 on dist-test-slave-jccs"
server_key: "0e45fb833f0e17b6d20c8625708898f8"
server_key_iv: "c7dcedf9c0a371e4f2ba37616b65bb9a"
server_key_version: "encryptionkey@0"
I20260928 04:52:38.600935 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/wal/instance:
uuid: "947a570d3172405e92adaa1c662b271a"
format_stamp: "Formatted at 2026-09-28 04:52:38 on dist-test-slave-jccs"
server_key: "0e45fb833f0e17b6d20c8625708898f8"
server_key_iv: "c7dcedf9c0a371e4f2ba37616b65bb9a"
server_key_version: "encryptionkey@0"
I20260928 04:52:38.602211 22673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 04:52:38.602882 22688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:38.603034 22673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:52:38.603116 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/wal
uuid: "947a570d3172405e92adaa1c662b271a"
format_stamp: "Formatted at 2026-09-28 04:52:38 on dist-test-slave-jccs"
server_key: "0e45fb833f0e17b6d20c8625708898f8"
server_key_iv: "c7dcedf9c0a371e4f2ba37616b65bb9a"
server_key_version: "encryptionkey@0"
I20260928 04:52:38.603168 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:38.618413 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } attempt: 2
I20260928 04:52:38.618856 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.618719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45161 (local address 127.0.0.1:33910)
0928 04:52:38.618778 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.618821 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 04:52:38.619236 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.619150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45161 (local address 127.0.0.1:33922)
0928 04:52:38.619196 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.619220 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:52:38.619352 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45161: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45161: connect: Connection refused (error 111)
I20260928 04:52:38.627750 22673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:38.629637 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:38.629779 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:38.633435 22673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45161
I20260928 04:52:38.633463 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45161 every 8 connection(s)
I20260928 04:52:38.633807 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
I20260928 04:52:38.634817 22741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:38.636336 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:38.643033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 22673
I20260928 04:52:38.643138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-1/wal/instance
I20260928 04:52:38.643328 392 external_mini_cluster.cc:1428] Setting key 246fd1a915243d9cf826ac0f5aa2b2d2
I20260928 04:52:38.644819 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.637113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.0.98.62:43655)
0928 04:52:38.637163 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.637165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.637304 (+ 139us) server_negotiation.cc:407] Connection header received
0928 04:52:38.637790 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.637792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.637801 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.637851 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.638412 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.638884 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.639783 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.639964 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.640012 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.642079 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.642080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.642080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.642087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.643556 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.643919 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.643921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.643921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.643984 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.644200 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.644200 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.644201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.644357 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.644421 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.644647 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.644731 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 04:52:38.644996 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.636967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43655 (local address 127.0.0.1:56664)
0928 04:52:38.637180 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.637241 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:38.637702 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:38.637929 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:38.637932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:38.638166 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:38.638343 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.638347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.638956 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.638957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.639716 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.639717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.640039 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.640039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.640141 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.640637 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:38.640645 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:38.641978 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:38.643631 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.643636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.643638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.643845 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.644049 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.644049 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.644050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.644139 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.644416 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:38.644419 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:38.644520 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 04:52:38.644622 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 04:52:38.644776 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
I20260928 04:52:38.645568 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 04:52:38.653157 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.645815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45161 (local address 127.0.0.1:33938)
0928 04:52:38.645862 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.645873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:38.645913 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:38.646164 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:38.646165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:38.646268 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:38.646314 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.646315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.647606 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.647607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.648084 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.648085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.648787 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.648788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.648840 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.649233 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:38.649238 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:38.650328 (+ 1090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:38.652256 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.652257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.652257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.652422 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.652629 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.652630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.652630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.652680 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.652913 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:38.652914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:38.652946 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 04:52:38.652995 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 04:52:38.653067 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 04:52:38.653348 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.645853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.0.98.61:45161)
0928 04:52:38.646015 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.646019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.646040 (+ 21us) server_negotiation.cc:407] Connection header received
0928 04:52:38.646061 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.646063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.646076 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.646153 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.646382 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.647515 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.648172 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.648700 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.648792 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.650408 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.650417 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.650417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.650424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.652180 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.652500 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.652503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.652504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.652565 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.652736 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.652737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.652737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.652858 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.652916 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.653034 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.653114 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260928 04:52:38.653946 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:38.654410 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.654151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47794)
0928 04:52:38.654335 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.654386 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
W20260928 04:52:38.654552 22690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
I20260928 04:52:38.654982 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.654875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47798)
0928 04:52:38.654922 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.654952 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:52:38.655100 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44655: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 04:52:38.668190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44655
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43655,127.0.98.61:45161,127.0.98.60:44655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:52:38.704339 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } attempt: 1
I20260928 04:52:38.704802 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.704668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47804)
0928 04:52:38.704732 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.704770 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 04:52:38.705154 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.705049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47820)
0928 04:52:38.705101 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.705126 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 04:52:38.705266 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44655: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
I20260928 04:52:38.710522 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } attempt: 3
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:52:38.717571 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.710797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45161 (local address 127.0.0.1:33950)
0928 04:52:38.710859 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.710870 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:38.710941 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:38.711074 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:38.711076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:38.711231 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:38.711295 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.711297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.711935 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.711937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.712543 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.712545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.712942 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.712943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.713027 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.713498 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:38.713503 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:38.714899 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:38.716531 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.716533 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.716533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.716702 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.716919 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.716920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.716920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.716983 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.717255 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:38.717256 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:38.717301 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 04:52:38.717386 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 04:52:38.717481 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 04:52:38.717597 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.710877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.0.98.61:45161)
0928 04:52:38.710937 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.710939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.710950 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:52:38.710994 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.710995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.711013 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.711070 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.711374 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.711851 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.712636 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.712848 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.712945 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.715001 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.715003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.715003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.715010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.716428 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.716774 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.716775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.716776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.716833 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.717046 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.717047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.717047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.717154 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.717261 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.717410 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.717482 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 04:52:38.718358 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:38.718750 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.718628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47824)
0928 04:52:38.718699 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.718728 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 04:52:38.719072 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.718980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47830)
0928 04:52:38.719026 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.719049 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:52:38.719163 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44655: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
W20260928 04:52:38.748694 22748 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:52:38.748934 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:38.748963 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:38.749006 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:38.750193 22748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:38.750258 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:38.750285 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:38.750317 22748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:38.750346 22748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:38.750315 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } attempt: 1
I20260928 04:52:38.750769 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.750637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47838)
0928 04:52:38.750697 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.750740 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 04:52:38.751127 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.751019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47852)
0928 04:52:38.751060 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.751084 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 04:52:38.751219 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44655: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
I20260928 04:52:38.752074 22748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43655,127.0.98.61:45161,127.0.98.60:44655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:38.752317 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:38.752542 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:38.757339 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:38.757332 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:38.757380 22748 server_base.cc:1034] running on GCE node
W20260928 04:52:38.757332 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:38.757700 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:38.757992 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:38.759150 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790571158759132 us; error 29 us; skew 500 ppm
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:38.761617 22748 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 04:52:38.762010 22748 webserver.cc:460] Webserver started at http://127.0.98.60:43349/ using document root <none> and password file <none>
I20260928 04:52:38.762228 22748 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:38.762292 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:38.762436 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:38.763479 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/data/instance:
uuid: "d4be46f93080434682634d78df3e795d"
format_stamp: "Formatted at 2026-09-28 04:52:38 on dist-test-slave-jccs"
server_key: "913d704fc9ccafbe2b2008a1cb44cb28"
server_key_iv: "94b4dc43382092bc24483d3b8bf9f047"
server_key_version: "encryptionkey@0"
I20260928 04:52:38.763835 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/wal/instance:
uuid: "d4be46f93080434682634d78df3e795d"
format_stamp: "Formatted at 2026-09-28 04:52:38 on dist-test-slave-jccs"
server_key: "913d704fc9ccafbe2b2008a1cb44cb28"
server_key_iv: "94b4dc43382092bc24483d3b8bf9f047"
server_key_version: "encryptionkey@0"
I20260928 04:52:38.765143 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:52:38.765901 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:38.766062 22748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:38.766168 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/wal
uuid: "d4be46f93080434682634d78df3e795d"
format_stamp: "Formatted at 2026-09-28 04:52:38 on dist-test-slave-jccs"
server_key: "913d704fc9ccafbe2b2008a1cb44cb28"
server_key_iv: "94b4dc43382092bc24483d3b8bf9f047"
server_key_version: "encryptionkey@0"
I20260928 04:52:38.766252 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:38.773396 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } attempt: 2
I20260928 04:52:38.773797 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.773676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47862)
0928 04:52:38.773731 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.773767 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 04:52:38.774142 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.774067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47876)
0928 04:52:38.774105 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.774127 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 04:52:38.774250 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44655: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44655: connect: Connection refused (error 111)
I20260928 04:52:38.781422 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:38.783612 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:38.783766 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:38.787518 22748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44655
I20260928 04:52:38.787545 22815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44655 every 8 connection(s)
I20260928 04:52:38.787865 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
I20260928 04:52:38.788798 22816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:38.790290 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:38.792478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 22748
I20260928 04:52:38.792549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790570897511889-392-0/minicluster-data/master-2/wal/instance
I20260928 04:52:38.792757 392 external_mini_cluster.cc:1428] Setting key bb175a65e3e68594010a228be16ee102
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:52:38.799924 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.790888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.0.98.62:43655)
0928 04:52:38.790950 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.790952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.791190 (+ 238us) server_negotiation.cc:407] Connection header received
0928 04:52:38.791655 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.791656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.791664 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.791716 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.792243 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.792734 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.794270 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.794558 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.794640 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.797066 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.797068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.797068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.797076 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.798420 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.798849 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.798850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.798850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.798912 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.799177 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.799179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.799179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.799317 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.799386 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.799727 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.799821 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 04:52:38.800060 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.790872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43655 (local address 127.0.0.1:56680)
0928 04:52:38.791060 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.791102 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:38.791554 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:38.791778 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:38.791781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:38.792005 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:38.792180 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.792184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.792836 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.792838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.794180 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.794183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.794664 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.794665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.794761 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.795238 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:38.795245 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:38.796977 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:38.798530 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.798535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.798537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.798770 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.798979 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.798980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.798981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.799105 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.799378 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:38.799397 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:38.799557 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 04:52:38.799691 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 04:52:38.799829 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260928 04:52:38.800551 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:38.801918 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.793206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.0.98.60:44655)
0928 04:52:38.793376 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.793379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.793397 (+ 18us) server_negotiation.cc:407] Connection header received
0928 04:52:38.793473 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.793477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.793492 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.793554 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.794267 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.795223 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.796160 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.796642 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.796695 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.798873 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.798879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.798879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.798885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.800737 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.801108 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.801111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.801111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.801192 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.801393 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.801394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.801395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.801533 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.801699 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.801742 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.801815 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:38.807528 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.800853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45161 (local address 127.0.0.1:33966)
0928 04:52:38.800976 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.800986 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:38.801034 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:38.801219 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:38.801220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:38.801333 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:38.801358 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.801359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.801959 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.801960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.802685 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.802687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.803095 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.803096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.803153 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.803552 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:38.803557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:38.804629 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:38.806545 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.806546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.806546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.806722 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.806935 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.806936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.806936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.807000 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.807260 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:38.807261 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:38.807303 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 04:52:38.807374 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 04:52:38.807455 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 04:52:38.807600 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.800925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.0.98.61:45161)
0928 04:52:38.800988 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.800990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.801020 (+ 30us) server_negotiation.cc:407] Connection header received
0928 04:52:38.801109 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.801109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.801119 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.801191 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.801435 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.801890 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.802783 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.803012 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.803093 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.804715 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.804717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.804717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.804723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.806455 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.806798 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.806799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.806800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.806856 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.807074 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.807074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.807074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.807214 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.807260 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.807408 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.807484 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 04:52:38.808229 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:38.809878 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.802764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.0.98.62:43655)
0928 04:52:38.802797 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.802799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.802806 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:52:38.802862 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.802863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.802879 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.802934 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.803368 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.803798 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.804425 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.804622 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.804701 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.806968 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.806969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.806970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.806976 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.808742 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.809077 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.809078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.809078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.809142 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.809348 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.809349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.809349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.809467 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.809644 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.809680 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.809748 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:52:38.815501 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.808456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47892)
0928 04:52:38.808498 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.808507 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:38.808545 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:38.808673 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:38.808674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:38.808773 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:38.808823 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.808825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.809649 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.809650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.810361 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.810362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.810826 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.810827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.810874 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.811236 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:38.811241 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:38.812514 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:38.814153 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.814154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.814154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.814356 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.814562 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.814563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.814563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.814611 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.815198 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:38.815198 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:38.815231 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 04:52:38.815304 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:52:38.815396 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 04:52:38.815551 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.808504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.0.98.60:44655)
0928 04:52:38.808543 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.808544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.808553 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:52:38.808600 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.808600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.808611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.808673 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.808896 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.809570 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.810558 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.810748 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.810825 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.812592 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.812593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.812593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.812599 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.814074 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.814433 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.814434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.814434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.814493 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.814669 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.814669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.814670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.814789 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.814854 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.815338 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.815441 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":20}
I20260928 04:52:38.817251 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d: Bootstrap starting.
I20260928 04:52:38.817986 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.810432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.0.98.61:45161)
0928 04:52:38.810464 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.810466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.810490 (+ 24us) server_negotiation.cc:407] Connection header received
0928 04:52:38.810546 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.810547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.810554 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.810617 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.811087 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.811775 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.812688 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.812965 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.813030 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.815194 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.815195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.815195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.815200 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.816846 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.817161 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.817162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.817163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.817242 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.817453 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.817454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.817454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.817596 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.817772 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.817818 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.817896 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 04:52:38.818255 22816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:38.818580 22816 log.cc:824] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:38.819401 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d: No bootstrap required, opened a new log
I20260928 04:52:38.819514 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:38.820839 22816 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } }
I20260928 04:52:38.820996 22816 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4be46f93080434682634d78df3e795d, State: Initialized, Role: FOLLOWER
I20260928 04:52:38.821090 22816 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } }
I20260928 04:52:38.820485 22640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8d9c07988d114f9c967190d7d6e95097"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56688
I20260928 04:52:38.821379 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } } }
I20260928 04:52:38.821436 22816 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:38.821477 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:52:38.823159 22833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:38.823210 22833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:38.823444 22748 master_runner.cc:186] Error getting master registration for 127.0.98.62:43655: OK, instance_id { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" instance_seqno: 1790571158481399 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:52:38.826277 22748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:52:38.826332 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } attempt: 2
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:52:38.833910 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.826660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.0.98.60:44655)
0928 04:52:38.826707 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.826709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.826762 (+ 53us) server_negotiation.cc:407] Connection header received
0928 04:52:38.826834 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.826835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.826842 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.826880 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.827199 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.827903 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.828865 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.829173 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.829254 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.831095 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.831097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.831097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.831104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.832705 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.833088 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.833089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.833089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.833144 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.833385 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.833385 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.833386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.833530 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.833603 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.833748 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.833821 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 04:52:38.833909 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.826613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47904)
0928 04:52:38.826709 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.826723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:38.826778 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:38.826894 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:38.826911 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:38.827083 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:38.827128 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.827130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.827992 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.827994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.828785 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.828788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.829265 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.829266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.829329 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.829771 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:38.829776 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:38.831010 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:38.832792 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.832793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.832793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.833014 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.833252 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.833253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.833254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.833318 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.833603 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:38.833604 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:38.833649 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 04:52:38.833722 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:52:38.833810 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 04:52:38.835438 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097: Bootstrap starting.
I20260928 04:52:38.836162 22666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:38.836462 22666 log.cc:824] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:38.837266 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097: No bootstrap required, opened a new log
I20260928 04:52:38.838603 22666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } }
I20260928 04:52:38.838737 22666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d9c07988d114f9c967190d7d6e95097, State: Initialized, Role: FOLLOWER
I20260928 04:52:38.838806 22666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } }
I20260928 04:52:38.839067 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } } }
I20260928 04:52:38.839113 22666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:38.839198 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:52:38.839145 22640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 04:52:38.840505 22640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:38.841105 22640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } }
I20260928 04:52:38.841645 22790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d9c07988d114f9c967190d7d6e95097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d4be46f93080434682634d78df3e795d"
I20260928 04:52:38.841765 22790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:38.842644 22790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d9c07988d114f9c967190d7d6e95097 in term 1.
I20260928 04:52:38.842811 22640 leader_election.cc:288] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [CANDIDATE]: Term 1 election: Requested vote from peers 947a570d3172405e92adaa1c662b271a (127.0.98.61:45161), d4be46f93080434682634d78df3e795d (127.0.98.60:44655)
I20260928 04:52:38.842875 22615 leader_election.cc:302] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d9c07988d114f9c967190d7d6e95097, d4be46f93080434682634d78df3e795d; no voters:
I20260928 04:52:38.842886 22598 master_runner.cc:186] Error getting master registration for 127.0.98.61:45161: OK, instance_id { permanent_uuid: "947a570d3172405e92adaa1c662b271a" instance_seqno: 1790571158632292 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:52:38.845620 22598 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:52:38.845680 22836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:38.845803 22836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [term 1 LEADER]: Becoming Leader. State: Replica: 8d9c07988d114f9c967190d7d6e95097, State: Running, Role: LEADER
I20260928 04:52:38.845883 22836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } }
I20260928 04:52:38.846271 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d9c07988d114f9c967190d7d6e95097. Latest consensus state: current_term: 1 leader_uuid: "8d9c07988d114f9c967190d7d6e95097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } } }
I20260928 04:52:38.846354 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:38.853315 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.844232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.0.98.60:44655)
0928 04:52:38.844267 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.844269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.844579 (+ 310us) server_negotiation.cc:407] Connection header received
0928 04:52:38.844648 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.844649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.844671 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.844714 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.846640 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.847366 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.848597 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.848890 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.848927 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.850372 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.850374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.850374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.850381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.852007 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.852435 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.852436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.852437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.852489 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.852688 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.852689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.852690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.852862 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.852964 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.853098 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.853203 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 04:52:38.853710 22847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:38.853765 22847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:38.854727 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.843892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.0.98.62:43655)
0928 04:52:38.843929 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.843932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.844449 (+ 517us) server_negotiation.cc:407] Connection header received
0928 04:52:38.844501 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.844502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.844514 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.844565 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.846661 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.847379 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.848514 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.848783 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.848820 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.851458 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.851459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.851459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.851465 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.853339 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.853735 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.853736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.853737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.853792 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.854065 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.854066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.854066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.854212 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.854296 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.854531 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.854626 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:52:38.855520 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.845228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.0.98.61:45161)
0928 04:52:38.845288 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.845290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.845299 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:52:38.845320 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.845321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.845331 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.845390 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.846060 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.846804 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.848828 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.849110 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.849150 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.850871 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.850873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.850873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.850879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.854108 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.854686 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.854687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.854687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.854741 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.854944 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.854945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.854945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.855065 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.855155 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.855342 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.855430 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:52:38.861783 22857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:38.861944 22857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:38.864128 22790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [term 1 FOLLOWER]: Refusing update from remote peer 8d9c07988d114f9c967190d7d6e95097: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:38.864445 22855 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:38.865391 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } attempt: 3
I20260928 04:52:38.865774 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d9c07988d114f9c967190d7d6e95097. Latest consensus state: current_term: 1 leader_uuid: "8d9c07988d114f9c967190d7d6e95097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } } }
I20260928 04:52:38.865846 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 04:52:38 dist-test-slave-jccs krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:52:38.870793 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d9c07988d114f9c967190d7d6e95097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } } }
I20260928 04:52:38.870882 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:38.871354 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d9c07988d114f9c967190d7d6e95097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } } }
I20260928 04:52:38.871438 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:52:38.872179 22859 mvcc.cc:205] Tried to move back new op lower bound from 7334179466702807040 to 7334179466633895936. Current Snapshot: MvccSnapshot[applied={T|T < 7334179466702807040}]
I20260928 04:52:38.872292 22860 mvcc.cc:205] Tried to move back new op lower bound from 7334179466702807040 to 7334179466633895936. Current Snapshot: MvccSnapshot[applied={T|T < 7334179466702807040}]
I20260928 04:52:38.873987 22857 catalog_manager.cc:1347] Generated new cluster ID: 16ca2ff0dc524f19ac35f13d05a09c33
I20260928 04:52:38.874039 22857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:38.875798 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.865650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44655 (local address 127.0.0.1:47926)
0928 04:52:38.865692 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:38.865703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:38.865737 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:38.866071 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:38.866072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:38.866235 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:38.866277 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.866279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.867169 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.867170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.867989 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:38.867992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.868535 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.868536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:38.868619 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.869095 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:38.869101 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:38.871016 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:38.874465 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.874466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.874467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.874719 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.875009 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:38.875010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:38.875010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:38.875080 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:38.875492 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:38.875493 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:38.875548 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 04:52:38.875604 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 04:52:38.875698 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 04:52:38.876561 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:38.865912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.0.98.60:44655)
0928 04:52:38.865955 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.865957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.865964 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:52:38.865982 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.865983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.865991 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.866050 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.866358 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.867087 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.868103 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:38.868409 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:38.868469 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:38.871214 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:38.871215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:38.871215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:38.871222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:38.874326 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.874812 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.874813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.874813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.874899 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:38.875164 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:38.875165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:38.875165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:38.875380 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:38.876214 (+ 834us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:38.876286 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 04:52:38.876418 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 04:52:38.878199 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a: Bootstrap starting.
I20260928 04:52:38.878902 22741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:38.879169 22741 log.cc:824] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:38.880208 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a: No bootstrap required, opened a new log
I20260928 04:52:38.882181 22741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } }
I20260928 04:52:38.882304 22741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947a570d3172405e92adaa1c662b271a, State: Initialized, Role: FOLLOWER
I20260928 04:52:38.882310 22857 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:38.882377 22741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } }
I20260928 04:52:38.882665 22741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:38.882958 22857 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:38.883985 22715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d9c07988d114f9c967190d7d6e95097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "947a570d3172405e92adaa1c662b271a"
I20260928 04:52:38.883919 22714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:38.885099 22714 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [term 1 FOLLOWER]: Refusing update from remote peer 8d9c07988d114f9c967190d7d6e95097: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:38.885375 22836 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [LEADER]: Connected to new peer: Peer: permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:38.885936 22715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d9c07988d114f9c967190d7d6e95097 in term 1.
I20260928 04:52:38.887987 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d9c07988d114f9c967190d7d6e95097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } } }
I20260928 04:52:38.888129 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:52:38.888221 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d9c07988d114f9c967190d7d6e95097. Latest consensus state: current_term: 1 leader_uuid: "8d9c07988d114f9c967190d7d6e95097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } } }
I20260928 04:52:38.888311 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:52:38.888379 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d9c07988d114f9c967190d7d6e95097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } } }
I20260928 04:52:38.888464 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:52:38.889580 22879 mvcc.cc:205] Tried to move back new op lower bound from 7334179466776453120 to 7334179466633895936. Current Snapshot: MvccSnapshot[applied={T|T < 7334179466702807040}]
W20260928 04:52:38.893688 22877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:38.893743 22877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:38.899333 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d9c07988d114f9c967190d7d6e95097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d9c07988d114f9c967190d7d6e95097" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } peers { permanent_uuid: "947a570d3172405e92adaa1c662b271a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45161 } } peers { permanent_uuid: "d4be46f93080434682634d78df3e795d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44655 } } }
I20260928 04:52:38.899425 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:38.901314 22857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d9c07988d114f9c967190d7d6e95097: Generated new TSK 0
I20260928 04:52:38.901471 22857 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:52:38.928557 22670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:38.927079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.0.98.62:43655)
0928 04:52:38.927123 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.927125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.927935 (+ 810us) server_negotiation.cc:407] Connection header received
0928 04:52:38.928047 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.928048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.928060 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.928092 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.928492 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56702: BlockingRecv error: recv got EOF from 127.0.0.1:56702 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 04:52:38.929234 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:38.927421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.0.98.60:44655)
0928 04:52:38.927458 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.927460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.928769 (+ 1309us) server_negotiation.cc:407] Connection header received
0928 04:52:38.928855 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.928856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.928865 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.928893 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.929161 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47932: BlockingRecv error: recv got EOF from 127.0.0.1:47932 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 04:52:38.929876 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:38.927706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.0.98.61:45161)
0928 04:52:38.927742 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.927744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.929429 (+ 1685us) server_negotiation.cc:407] Connection header received
0928 04:52:38.929516 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.929517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.929527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.929555 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.929823 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33990: BlockingRecv error: recv got EOF from 127.0.0.1:33990 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 04:52:38.931931 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:38.931296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.0.98.61:45161)
0928 04:52:38.931326 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.931328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.931476 (+ 148us) server_negotiation.cc:407] Connection header received
0928 04:52:38.931570 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.931571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.931587 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.931614 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.931892 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33992: BlockingRecv error: recv got EOF from 127.0.0.1:33992 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 04:52:38.934715 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:38.934073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.0.98.61:45161)
0928 04:52:38.934101 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.934102 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.934286 (+ 184us) server_negotiation.cc:407] Connection header received
0928 04:52:38.934367 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.934368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.934376 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.934401 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.934679 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33998: BlockingRecv error: recv got EOF from 127.0.0.1:33998 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 04:52:38.938416 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:38.937750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.0.98.61:45161)
0928 04:52:38.937787 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.937789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.937969 (+ 180us) server_negotiation.cc:407] Connection header received
0928 04:52:38.938068 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.938069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.938078 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.938105 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.938379 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34010: BlockingRecv error: recv got EOF from 127.0.0.1:34010 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 04:52:38.950732 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:38.950002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.0.98.61:45161)
0928 04:52:38.950047 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.950049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.950248 (+ 199us) server_negotiation.cc:407] Connection header received
0928 04:52:38.950337 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.950338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.950348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.950377 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.950684 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34022: BlockingRecv error: recv got EOF from 127.0.0.1:34022 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 04:52:38.969890 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:38.965411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.0.98.61:45161)
0928 04:52:38.965448 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.965450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.968464 (+ 3014us) server_negotiation.cc:407] Connection header received
0928 04:52:38.968538 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.968539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.968551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.968618 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.969834 (+ 1216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34036: BlockingRecv error: recv got EOF from 127.0.0.1:34036 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 04:52:38.997642 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:38.987042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.0.98.61:45161)
0928 04:52:38.987079 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:38.987081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:38.991528 (+ 4447us) server_negotiation.cc:407] Connection header received
0928 04:52:38.991603 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:38.991604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:38.991616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:38.991682 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:38.997585 (+ 5903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34038: BlockingRecv error: recv got EOF from 127.0.0.1:34038 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 04:52:39.034118 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:39.031682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.0.98.61:45161)
0928 04:52:39.031741 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:39.031743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:39.033462 (+ 1719us) server_negotiation.cc:407] Connection header received
0928 04:52:39.033609 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:39.033611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:39.033629 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:39.033666 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:39.034053 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34044: BlockingRecv error: recv got EOF from 127.0.0.1:34044 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 04:52:39.100390 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:39.099583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.0.98.61:45161)
0928 04:52:39.099716 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:39.099719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:39.099796 (+ 77us) server_negotiation.cc:407] Connection header received
0928 04:52:39.099896 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:39.099898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:39.099909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:39.099981 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:39.100317 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34058: BlockingRecv error: recv got EOF from 127.0.0.1:34058 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260928 04:52:39.230350 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:39.229632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.0.98.61:45161)
0928 04:52:39.229706 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:39.229708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:39.229822 (+ 114us) server_negotiation.cc:407] Connection header received
0928 04:52:39.229888 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:39.229889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:39.229901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:39.229942 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:39.230276 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34072: BlockingRecv error: recv got EOF from 127.0.0.1:34072 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 04:52:39.488000 22745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:39.487407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.0.98.61:45161)
0928 04:52:39.487472 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:39.487474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:39.487505 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:52:39.487562 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:39.487563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:39.487571 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:39.487630 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:39.487931 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34086: BlockingRecv error: recv got EOF from 127.0.0.1:34086 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 04:52:39.823603 22833 catalog_manager.cc:1259] Loaded cluster ID: 16ca2ff0dc524f19ac35f13d05a09c33
I20260928 04:52:39.823670 22833 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d: loading cluster ID for follower catalog manager: success
I20260928 04:52:39.824457 22833 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d: acquiring CA information for follower catalog manager: success
I20260928 04:52:39.824772 22833 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d4be46f93080434682634d78df3e795d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:52:39.894111 22877 catalog_manager.cc:1259] Loaded cluster ID: 16ca2ff0dc524f19ac35f13d05a09c33
I20260928 04:52:39.894217 22877 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a: loading cluster ID for follower catalog manager: success
I20260928 04:52:39.895042 22877 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a: acquiring CA information for follower catalog manager: success
I20260928 04:52:39.895367 22877 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 947a570d3172405e92adaa1c662b271a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:52:40.003911 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:40.001297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.0.98.61:45161)
0928 04:52:40.001437 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:40.001439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:40.001449 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:52:40.001526 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:40.001527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:40.001593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:40.001680 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:40.001819 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:40.002317 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:40.003090 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:40.003297 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:40.003370 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:40.003550 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:40.003662 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:40.003751 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:40.003788 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 04:52:40.003807 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260928 04:52:40.007988 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:40.005754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.0.98.60:44655)
0928 04:52:40.005861 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:40.005863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:40.005872 (+ 9us) server_negotiation.cc:407] Connection header received
0928 04:52:40.006014 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:40.006014 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:40.006044 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:40.006110 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:40.006212 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:40.006660 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:40.007381 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:40.007566 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:40.007617 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:40.007738 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:40.007827 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:40.007884 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:40.007914 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 04:52:40.007930 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":69,"threads_started":1}
I20260928 04:52:40.012142 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:40.009704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.0.98.60:44655)
0928 04:52:40.009755 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:40.009757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:40.009765 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:52:40.009818 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:40.009819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:40.009827 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:40.009878 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:40.010014 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:40.010481 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:40.011375 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:40.011660 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:40.011717 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:40.011891 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:40.011921 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:40.011978 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:40.012045 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 04:52:40.012061 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 04:52:40.012632 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:40.009535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.0.98.61:45161)
0928 04:52:40.009570 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:40.009572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:40.009763 (+ 191us) server_negotiation.cc:407] Connection header received
0928 04:52:40.009863 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:40.009864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:40.009875 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:40.009952 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:40.010069 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:40.010786 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:40.011699 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:40.011928 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:40.011990 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:40.012381 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:40.012412 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:40.012460 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:40.012525 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 04:52:40.012547 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:52:40.012795 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:40.009512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.0.98.62:43655)
0928 04:52:40.009633 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:40.009635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:40.009646 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:52:40.009758 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:40.009760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:40.009810 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:40.009884 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:40.010034 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:40.010796 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:40.011712 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:40.012040 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:40.012102 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:40.012391 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:40.012488 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:40.012541 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:40.012643 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 04:52:40.012670 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":44,"threads_started":1}
I20260928 04:52:40.014528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 22598
I20260928 04:52:40.019544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 22673
I20260928 04:52:40.024701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 22748
2026-09-28T04:52:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2796 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 04:52:40 dist-test-slave-jccs krb5kdc[22966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 04:52:40 dist-test-slave-jccs krb5kdc[22966](info): set up 2 sockets
Sep 28 04:52:40 dist-test-slave-jccs krb5kdc[22966](info): commencing operation
krb5kdc: starting...
W20260928 04:52:41.057353 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/test-user.keytab.
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T04:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T04:52:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 04:52:41.171813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35763,127.0.98.61:42467,127.0.98.60:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 04:52:41.250265 22979 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:52:41.250507 22979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:41.250542 22979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:41.250584 22979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:41.251719 22979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:41.251785 22979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:41.251807 22979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:41.251830 22979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:41.251858 22979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:41.253309 22979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35763,127.0.98.61:42467,127.0.98.60:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:41.253611 22979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:41.253824 22979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:41.258343 22984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:41.258315 22987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:41.258319 22985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:41.258504 22979 server_base.cc:1034] running on GCE node
I20260928 04:52:41.258852 22979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:41.259155 22979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:41.260310 22979 hybrid_clock.cc:648] HybridClock initialized: now 1790571161260295 us; error 27 us; skew 500 ppm
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:41.262674 22979 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 04:52:41.263062 22979 webserver.cc:460] Webserver started at http://127.0.98.62:36295/ using document root <none> and password file <none>
I20260928 04:52:41.263276 22979 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:41.263342 22979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:41.263489 22979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:41.264437 22979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/data/instance:
uuid: "18c3d522ae7b4f618de650b3ba641166"
format_stamp: "Formatted at 2026-09-28 04:52:41 on dist-test-slave-jccs"
server_key: "e460d0b110c3b82cb4a0d30a83885b70"
server_key_iv: "99f2df931e4d78f4f8da242d0688d414"
server_key_version: "encryptionkey@0"
I20260928 04:52:41.264761 22979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/wal/instance:
uuid: "18c3d522ae7b4f618de650b3ba641166"
format_stamp: "Formatted at 2026-09-28 04:52:41 on dist-test-slave-jccs"
server_key: "e460d0b110c3b82cb4a0d30a83885b70"
server_key_iv: "99f2df931e4d78f4f8da242d0688d414"
server_key_version: "encryptionkey@0"
I20260928 04:52:41.266098 22979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 04:52:41.266819 22994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:41.266976 22979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:41.267069 22979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/wal
uuid: "18c3d522ae7b4f618de650b3ba641166"
format_stamp: "Formatted at 2026-09-28 04:52:41 on dist-test-slave-jccs"
server_key: "e460d0b110c3b82cb4a0d30a83885b70"
server_key_iv: "99f2df931e4d78f4f8da242d0688d414"
server_key_version: "encryptionkey@0"
I20260928 04:52:41.267141 22979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:41.275022 22979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:41.276640 22979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:41.276778 22979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:41.280537 22979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35763
I20260928 04:52:41.280583 23046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35763 every 8 connection(s)
I20260928 04:52:41.280889 22979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/data/info.pb
I20260928 04:52:41.281863 23047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:41.283806 23047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:41.286823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 22979
I20260928 04:52:41.286938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-0/wal/instance
I20260928 04:52:41.287115 392 external_mini_cluster.cc:1428] Setting key ce4afa9b3ae992069e8af920a9a2715a
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:41.293411 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.284491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35763 (local address 127.0.0.1:41390)
0928 04:52:41.284651 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.284683 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:41.285226 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:41.285512 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:41.285514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:41.285783 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:41.285928 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.285931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.286916 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.286917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.287655 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.287657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.288157 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.288158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.288225 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.288791 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:41.288799 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:41.290223 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:41.291926 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.291931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.291934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.292122 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.292374 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.292375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.292375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.292460 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.292807 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:41.292812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:41.292870 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 04:52:41.292975 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 04:52:41.293229 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":98,"thread_start_us":78,"threads_started":1}
I20260928 04:52:41.293411 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.284651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.0.98.62:35763)
0928 04:52:41.284764 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.284769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.284800 (+ 31us) server_negotiation.cc:407] Connection header received
0928 04:52:41.285389 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.285391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.285415 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.285508 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.286042 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.286823 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.287756 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.288049 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.288158 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.290318 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.290325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.290325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.290331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.291832 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.292217 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.292219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.292220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.292296 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.292533 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.292533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.292534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.292733 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.292828 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.293013 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.293146 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":45,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 04:52:41.294189 23047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:41.294588 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.294433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42467 (local address 127.0.0.1:49622)
0928 04:52:41.294492 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.294558 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 04:52:41.294737 22997 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42467: connect: Connection refused (error 111)
I20260928 04:52:41.295212 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.295125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42467 (local address 127.0.0.1:49630)
0928 04:52:41.295170 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.295195 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 04:52:41.295327 23047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42467: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 04:52:41.310319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42467
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35763,127.0.98.61:42467,127.0.98.60:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:52:41.344518 23047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } attempt: 1
I20260928 04:52:41.345013 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.344881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42467 (local address 127.0.0.1:49634)
0928 04:52:41.344940 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.344981 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 04:52:41.345366 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.345289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42467 (local address 127.0.0.1:49650)
0928 04:52:41.345318 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.345341 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 04:52:41.345475 23047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42467: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42467: connect: Connection refused (error 111)
W20260928 04:52:41.389288 23054 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:52:41.389535 23054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:41.389564 23054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:41.389606 23054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:41.390686 23054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:41.390749 23054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:41.390774 23054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:41.390805 23054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:41.390829 23054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:41.392202 23054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35763,127.0.98.61:42467,127.0.98.60:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:41.392499 23054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:41.392710 23054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:41.397400 23062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:41.397504 23059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:41.397524 23060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:41.397519 23054 server_base.cc:1034] running on GCE node
I20260928 04:52:41.397850 23054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:41.398118 23054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:41.399255 23054 hybrid_clock.cc:648] HybridClock initialized: now 1790571161399240 us; error 29 us; skew 500 ppm
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:41.401508 23054 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 04:52:41.402009 23054 webserver.cc:460] Webserver started at http://127.0.98.61:34645/ using document root <none> and password file <none>
I20260928 04:52:41.402253 23054 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:41.402319 23054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:41.402469 23054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:41.403441 23054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/data/instance:
uuid: "2c99e194a5d34b13a128cc0daae5de2e"
format_stamp: "Formatted at 2026-09-28 04:52:41 on dist-test-slave-jccs"
server_key: "a79e5fbbd18cd84b8becc6af931411b9"
server_key_iv: "24b52846bddf6d53813c275b0dd06cfd"
server_key_version: "encryptionkey@0"
I20260928 04:52:41.403744 23054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/wal/instance:
uuid: "2c99e194a5d34b13a128cc0daae5de2e"
format_stamp: "Formatted at 2026-09-28 04:52:41 on dist-test-slave-jccs"
server_key: "a79e5fbbd18cd84b8becc6af931411b9"
server_key_iv: "24b52846bddf6d53813c275b0dd06cfd"
server_key_version: "encryptionkey@0"
I20260928 04:52:41.404834 23054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 04:52:41.405475 23069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:41.405617 23054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 04:52:41.405683 23054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/wal
uuid: "2c99e194a5d34b13a128cc0daae5de2e"
format_stamp: "Formatted at 2026-09-28 04:52:41 on dist-test-slave-jccs"
server_key: "a79e5fbbd18cd84b8becc6af931411b9"
server_key_iv: "24b52846bddf6d53813c275b0dd06cfd"
server_key_version: "encryptionkey@0"
I20260928 04:52:41.405731 23054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:41.413625 23047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } attempt: 2
I20260928 04:52:41.414016 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.413895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42467 (local address 127.0.0.1:49658)
0928 04:52:41.413952 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.413989 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 04:52:41.414331 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.414231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42467 (local address 127.0.0.1:49666)
0928 04:52:41.414276 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.414303 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:52:41.414455 23047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42467: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42467: connect: Connection refused (error 111)
I20260928 04:52:41.415572 23054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:41.417284 23054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:41.417425 23054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:41.421072 23054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42467
I20260928 04:52:41.421131 23121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42467 every 8 connection(s)
I20260928 04:52:41.421466 23054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/data/info.pb
I20260928 04:52:41.422363 23122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:41.423805 23122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:41.424646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 23054
I20260928 04:52:41.424760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-1/wal/instance
I20260928 04:52:41.424944 392 external_mini_cluster.cc:1428] Setting key 8db47591fba6f261a1c6ec85b93e3b93
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 04:52:41.433049 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.424451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.0.98.62:35763)
0928 04:52:41.424500 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.424502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.424690 (+ 188us) server_negotiation.cc:407] Connection header received
0928 04:52:41.425297 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.425298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.425312 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.425376 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.425946 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.426420 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.427694 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.427885 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.427945 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.430260 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.430261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.430262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.430268 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.431639 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.432071 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.432072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.432073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.432132 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.432414 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.432415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.432415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.432539 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.432609 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.432877 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.432960 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 04:52:41.433256 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.424430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35763 (local address 127.0.0.1:41394)
0928 04:52:41.424589 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.424620 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:41.425205 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:41.425436 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:41.425440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:41.425674 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:41.425846 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.425850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.426513 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.426515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.427609 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.427612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.427982 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.427983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.428134 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.428597 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:41.428607 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:41.430154 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:41.431734 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.431740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.431743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.431977 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.432200 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.432201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.432201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.432338 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.432612 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:41.432619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:41.432752 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 04:52:41.432856 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 04:52:41.433004 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260928 04:52:41.433806 23122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:52:41.441138 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.434045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42467 (local address 127.0.0.1:49672)
0928 04:52:41.434093 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.434104 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:41.434144 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:41.434388 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:41.434389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:41.434487 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:41.434532 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.434533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.435459 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.435460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.435922 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.435923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.436384 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.436385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.436444 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.436827 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:41.436832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:41.438215 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:41.440192 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.440193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.440193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.440334 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.440564 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.440565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.440565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.440611 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.440924 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:41.440925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:41.440955 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 04:52:41.441002 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 04:52:41.441080 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 04:52:41.441373 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.434122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.0.98.61:42467)
0928 04:52:41.434265 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.434269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.434289 (+ 20us) server_negotiation.cc:407] Connection header received
0928 04:52:41.434306 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.434308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.434317 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.434373 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.434590 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.435372 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.436022 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.436302 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.436387 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.438292 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.438299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.438299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.438305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.440111 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.440423 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.440426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.440426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.440496 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.440672 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.440673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.440673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.440854 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.440945 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.441037 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.441142 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260928 04:52:41.441999 23122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:41.442319 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.442217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40862)
0928 04:52:41.442249 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.442294 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 04:52:41.442512 23073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
I20260928 04:52:41.442930 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.442848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40868)
0928 04:52:41.442893 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.442915 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 04:52:41.443053 23122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44971: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 04:52:41.450085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNuqiey/build/release/bin/kudu
/tmp/dist-test-taskNuqiey/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44971
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35763,127.0.98.61:42467,127.0.98.60:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 04:52:41.492374 23122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } attempt: 1
I20260928 04:52:41.492846 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.492714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40878)
0928 04:52:41.492779 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.492818 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 04:52:41.493230 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.493090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40894)
0928 04:52:41.493133 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.493156 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 04:52:41.493309 23122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44971: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
I20260928 04:52:41.505600 23047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } attempt: 3
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:52:41.512864 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.506104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.0.98.61:42467)
0928 04:52:41.506160 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.506164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.506177 (+ 13us) server_negotiation.cc:407] Connection header received
0928 04:52:41.506204 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.506205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.506215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.506276 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.506639 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.507170 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.508190 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.508439 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.508569 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.510408 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.510410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.510411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.510418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.511727 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.512066 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.512067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.512068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.512122 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.512322 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.512323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.512323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.512455 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.512545 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.512696 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.512768 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 04:52:41.512864 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.506033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42467 (local address 127.0.0.1:49678)
0928 04:52:41.506099 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.506110 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:41.506154 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:41.506302 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:41.506303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:41.506495 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:41.506567 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.506569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.507265 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.507266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.508116 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.508119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.508569 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.508570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.508672 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.509119 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:41.509125 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:41.510321 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:41.511819 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.511820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.511820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.511979 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.512186 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.512187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.512187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.512241 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.512538 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:41.512539 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:41.512585 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 04:52:41.512673 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 04:52:41.512768 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 04:52:41.513235 23047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:41.513603 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.513492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40904)
0928 04:52:41.513552 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.513579 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 04:52:41.513999 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.513922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40918)
0928 04:52:41.513963 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.513985 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:52:41.514114 23047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44971: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
W20260928 04:52:41.531158 23129 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 04:52:41.531323 23129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 04:52:41.531338 23129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 04:52:41.531355 23129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 04:52:41.532362 23129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 04:52:41.532393 23129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 04:52:41.532407 23129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 04:52:41.532419 23129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 04:52:41.532431 23129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 04:52:41.533916 23129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35763,127.0.98.61:42467,127.0.98.60:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 03:57:00 UTC on ccycloud.quasar-rokiph.root.comops.site
build id 4676561
I20260928 04:52:41.534083 23129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 04:52:41.534233 23129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 04:52:41.538568 23135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 04:52:41.538563 23137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:41.538722 23129 server_base.cc:1034] running on GCE node
W20260928 04:52:41.538560 23134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 04:52:41.539026 23129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 04:52:41.539319 23129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 04:52:41.540470 23129 hybrid_clock.cc:648] HybridClock initialized: now 1790571161540461 us; error 37 us; skew 500 ppm
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 04:52:41.542809 23129 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 04:52:41.543257 23129 webserver.cc:460] Webserver started at http://127.0.98.60:39739/ using document root <none> and password file <none>
I20260928 04:52:41.543484 23129 fs_manager.cc:359] Metadata directory not provided
I20260928 04:52:41.543562 23129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 04:52:41.543706 23129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 04:52:41.544800 23129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/data/instance:
uuid: "529fdd30500d4296adcf61b1047c8714"
format_stamp: "Formatted at 2026-09-28 04:52:41 on dist-test-slave-jccs"
server_key: "701968b1b37764b04df125fcd290af60"
server_key_iv: "1cbb91dd6c9e66e98af1763f2bba9ea5"
server_key_version: "encryptionkey@0"
I20260928 04:52:41.545152 23129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/wal/instance:
uuid: "529fdd30500d4296adcf61b1047c8714"
format_stamp: "Formatted at 2026-09-28 04:52:41 on dist-test-slave-jccs"
server_key: "701968b1b37764b04df125fcd290af60"
server_key_iv: "1cbb91dd6c9e66e98af1763f2bba9ea5"
server_key_version: "encryptionkey@0"
I20260928 04:52:41.545229 23047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } attempt: 1
I20260928 04:52:41.545617 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.545508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40932)
0928 04:52:41.545563 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.545595 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 04:52:41.545919 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.545828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40942)
0928 04:52:41.545869 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.545892 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 04:52:41.546015 23047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44971: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
I20260928 04:52:41.546698 23129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 04:52:41.547374 23144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 04:52:41.547550 23129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 04:52:41.547642 23129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/wal
uuid: "529fdd30500d4296adcf61b1047c8714"
format_stamp: "Formatted at 2026-09-28 04:52:41 on dist-test-slave-jccs"
server_key: "701968b1b37764b04df125fcd290af60"
server_key_iv: "1cbb91dd6c9e66e98af1763f2bba9ea5"
server_key_version: "encryptionkey@0"
I20260928 04:52:41.547724 23129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 04:52:41.560458 23129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 04:52:41.561434 23122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } attempt: 2
I20260928 04:52:41.561828 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.561711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40952)
0928 04:52:41.561768 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.561803 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 04:52:41.562222 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.562144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40966)
0928 04:52:41.562185 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.562208 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 04:52:41.562358 23122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44971: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44971: connect: Connection refused (error 111)
I20260928 04:52:41.562829 23129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 04:52:41.562983 23129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 04:52:41.566537 23129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44971
I20260928 04:52:41.566625 23196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44971 every 8 connection(s)
I20260928 04:52:41.566905 23129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/data/info.pb
I20260928 04:52:41.567845 23197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 04:52:41.569350 23197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } has no permanent_uuid. Determining permanent_uuid...
I20260928 04:52:41.574358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNuqiey/build/release/bin/kudu as pid 23129
I20260928 04:52:41.574486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNuqiey/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790570897511889-392-0/minicluster-data/master-2/wal/instance
I20260928 04:52:41.574687 392 external_mini_cluster.cc:1428] Setting key 5a33429b995d4e9a67db0fd6f8ba854a
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 04:52:41.578483 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.570050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.0.98.62:35763)
0928 04:52:41.570137 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.570140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.570303 (+ 163us) server_negotiation.cc:407] Connection header received
0928 04:52:41.570767 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.570768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.570780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.570841 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.571445 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.571908 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.572901 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.573106 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.573220 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.575695 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.575696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.575697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.575705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.577166 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.577561 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.577562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.577562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.577622 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.577837 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.577838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.577838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.577997 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.578059 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.578332 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.578392 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 04:52:41.578706 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.570012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35763 (local address 127.0.0.1:41398)
0928 04:52:41.570181 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.570226 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:41.570667 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:41.570911 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:41.570919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:41.571174 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:41.571352 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.571355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.572015 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.572016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.572796 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.572798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.573239 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.573240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.573368 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.573879 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:41.573887 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:41.575614 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:41.577278 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.577283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.577284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.577488 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.577689 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.577689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.577689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.577778 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.578059 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:41.578063 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:41.578201 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 04:52:41.578311 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 04:52:41.578459 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260928 04:52:41.579257 23197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:52:41.584069 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.575075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.0.98.60:44971)
0928 04:52:41.575281 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.575286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.575312 (+ 26us) server_negotiation.cc:407] Connection header received
0928 04:52:41.575334 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.575336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.575356 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.575436 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.576171 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.577012 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.577986 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.578409 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.578473 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.581100 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.581106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.581106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.581111 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.582901 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.583265 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.583268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.583268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.583331 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.583539 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.583540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.583540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.583674 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.583813 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.583891 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.583963 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260928 04:52:41.585997 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.579606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.0.98.61:42467)
0928 04:52:41.579665 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.579666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.579710 (+ 44us) server_negotiation.cc:407] Connection header received
0928 04:52:41.579784 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.579785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.579793 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.579835 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.580139 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.580794 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.581564 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.581781 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.581834 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.583424 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.583425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.583425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.583434 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.584963 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.585356 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.585358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.585358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.585417 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.585563 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.585564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.585564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.585688 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.585762 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.585882 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.585935 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 04:52:41.586025 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.579531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42467 (local address 127.0.0.1:49684)
0928 04:52:41.579658 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.579669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:41.579726 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:41.579839 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:41.579840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:41.580001 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:41.580058 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.580060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.580865 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.580866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.581491 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.581493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.581851 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.581852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.581928 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.582201 (+ 273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:41.582206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:41.583339 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:41.585063 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.585063 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.585064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.585290 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.585483 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.585483 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.585483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.585526 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.585758 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:41.585759 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:41.585795 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 04:52:41.585861 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 04:52:41.585938 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 04:52:41.586681 23197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:52:41.592435 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.585064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.0.98.62:35763)
0928 04:52:41.585099 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.585101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.585107 (+ 6us) server_negotiation.cc:407] Connection header received
0928 04:52:41.585133 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.585134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.585143 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.585226 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.585773 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.586336 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.587185 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.587445 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.587512 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.590003 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.590004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.590004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.590010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.591383 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.591695 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.591696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.591697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.591749 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.591948 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.591949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.591949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.592054 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.592221 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.592256 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.592340 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 04:52:41.593641 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.586972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.0.98.60:44971)
0928 04:52:41.587006 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.587007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.587015 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:52:41.587032 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.587032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.587040 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.587095 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.587289 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.587958 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.588904 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.589246 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.589313 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.591082 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.591083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.591083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.591088 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.592383 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.592746 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.592747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.592747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.592821 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.593042 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.593043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.593043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.593209 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.593280 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.593433 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.593555 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":19}
I20260928 04:52:41.593683 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.586915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40984)
0928 04:52:41.586950 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.586958 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:41.586987 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:41.587103 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:41.587103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:41.587199 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:41.587224 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.587225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.588033 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.588034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.588828 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.588830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.589312 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.589313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.589348 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.589700 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:41.589705 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:41.591004 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:41.592493 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.592494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.592494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.592669 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.592887 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.592888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.592888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.592978 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.593287 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:41.593288 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:41.593334 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 04:52:41.593407 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 04:52:41.593511 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 04:52:41.595145 23197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714: Bootstrap starting.
I20260928 04:52:41.596130 23197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:41.596493 23197 log.cc:824] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:41.597347 23197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714: No bootstrap required, opened a new log
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 04:52:41.598800 23197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } }
I20260928 04:52:41.598945 23197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529fdd30500d4296adcf61b1047c8714, State: Initialized, Role: FOLLOWER
I20260928 04:52:41.599040 23197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } }
I20260928 04:52:41.599335 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } } }
I20260928 04:52:41.599385 23197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:41.599436 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:52:41.600775 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.593137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.0.98.61:42467)
0928 04:52:41.593196 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.593198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.593209 (+ 11us) server_negotiation.cc:407] Connection header received
0928 04:52:41.593261 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.593262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.593271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.593317 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.593886 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.594468 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.595337 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.595520 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.595570 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.598023 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.598025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.598025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.598031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.599612 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.600023 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.600024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.600025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.600078 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.600287 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.600288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.600288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.600394 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.600578 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.600612 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.600678 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 04:52:41.601115 23214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:41.601166 23214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:41.602211 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 04:52:41.603338 23021 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "18c3d522ae7b4f618de650b3ba641166"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41412
I20260928 04:52:41.601675 23129 master_runner.cc:186] Error getting master registration for 127.0.98.62:35763: OK, instance_id { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" instance_seqno: 1790571161279535 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:52:41.604696 23129 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 04:52:41.621109 23047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } attempt: 2
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:52:41.629264 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.621578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.0.98.60:44971)
0928 04:52:41.621643 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.621645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.621655 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:52:41.621683 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.621684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.621695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.621770 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.622124 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.622892 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.623862 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.624151 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.624223 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.626518 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.626519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.626520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.626526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.627952 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.628356 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.628357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.628357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.628435 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.628632 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.628632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.628633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.628811 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.628900 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.629058 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.629133 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 04:52:41.629274 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.621504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:40998)
0928 04:52:41.621550 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.621563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:41.621610 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:41.621792 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:41.621793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:41.621983 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:41.622035 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.622037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.622974 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.622975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.623773 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.623774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.624250 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.624252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.624320 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.624869 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:41.624874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:41.626398 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:41.628033 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.628034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.628035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.628249 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.628503 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.628504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.628504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.628567 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.628900 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:41.628901 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:41.628949 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 04:52:41.629025 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 04:52:41.629122 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 04:52:41.630959 23047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166: Bootstrap starting.
I20260928 04:52:41.631681 23047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:41.631985 23047 log.cc:824] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:41.632843 23047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166: No bootstrap required, opened a new log
I20260928 04:52:41.634217 23047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } }
I20260928 04:52:41.634366 23047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c3d522ae7b4f618de650b3ba641166, State: Initialized, Role: FOLLOWER
I20260928 04:52:41.634466 23047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } }
I20260928 04:52:41.634768 23047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:41.634734 23217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } } }
I20260928 04:52:41.634825 23217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:52:41.634918 23021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 04:52:41.634976 23021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:41.635679 23021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } }
I20260928 04:52:41.636570 23171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18c3d522ae7b4f618de650b3ba641166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "529fdd30500d4296adcf61b1047c8714"
I20260928 04:52:41.636687 23171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:41.637377 23171 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18c3d522ae7b4f618de650b3ba641166 in term 1.
I20260928 04:52:41.637596 22998 leader_election.cc:302] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c3d522ae7b4f618de650b3ba641166, 529fdd30500d4296adcf61b1047c8714; no voters:
I20260928 04:52:41.637724 23217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 04:52:41.637756 23021 leader_election.cc:288] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [CANDIDATE]: Term 1 election: Requested vote from peers 2c99e194a5d34b13a128cc0daae5de2e (127.0.98.61:42467), 529fdd30500d4296adcf61b1047c8714 (127.0.98.60:44971)
I20260928 04:52:41.637871 23217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [term 1 LEADER]: Becoming Leader. State: Replica: 18c3d522ae7b4f618de650b3ba641166, State: Running, Role: LEADER
I20260928 04:52:41.637981 23217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } }
I20260928 04:52:41.638346 23217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18c3d522ae7b4f618de650b3ba641166. Latest consensus state: current_term: 1 leader_uuid: "18c3d522ae7b4f618de650b3ba641166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } } }
I20260928 04:52:41.638410 23217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:41.640317 22979 master_runner.cc:186] Error getting master registration for 127.0.98.61:42467: OK, instance_id { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" instance_seqno: 1790571161420126 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 04:52:41.643111 22979 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 04:52:41.636445 23228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:41.643260 23228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:41.643425 23232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 04:52:41.644006 23232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 04:52:41.652736 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.640717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.0.98.60:44971)
0928 04:52:41.640752 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.640754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.644847 (+ 4093us) server_negotiation.cc:407] Connection header received
0928 04:52:41.644947 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.644947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.644956 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.644983 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.645700 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.646363 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.647343 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.647633 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.647673 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.649243 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.649245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.649245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.649251 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.650987 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.651578 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.651579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.651580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.651635 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.651961 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.651977 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.651977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.652116 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.652204 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.652549 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.652629 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:52:41.653848 23122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } attempt: 3
I20260928 04:52:41.654234 23171 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [term 1 FOLLOWER]: Refusing update from remote peer 18c3d522ae7b4f618de650b3ba641166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:41.654829 23230 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:41.656487 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18c3d522ae7b4f618de650b3ba641166. Latest consensus state: current_term: 1 leader_uuid: "18c3d522ae7b4f618de650b3ba641166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } } }
I20260928 04:52:41.656576 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 04:52:41 dist-test-slave-jccs krb5kdc[22966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790571161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 04:52:41.662756 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.640282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.0.98.62:35763)
0928 04:52:41.640316 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.640318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.641586 (+ 1268us) server_negotiation.cc:407] Connection header received
0928 04:52:41.641794 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.641795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.641815 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.641844 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.644638 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.645398 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.647886 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.648545 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.648593 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.652841 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.652842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.652843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.652850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.655457 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.659410 (+ 3953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.659411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.659411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.659480 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.661669 (+ 2189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.661670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.661671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.661807 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.661895 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.662563 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.662642 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 04:52:41.663125 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.639864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.0.98.61:42467)
0928 04:52:41.639906 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.639908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.641062 (+ 1154us) server_negotiation.cc:407] Connection header received
0928 04:52:41.641196 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.641197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.641209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.641240 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.643611 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.644329 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.649102 (+ 4773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.649746 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.649806 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.655618 (+ 5812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.655619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.655619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.655636 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.657317 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.659561 (+ 2244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.659562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.659562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.659627 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.661952 (+ 2325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.661952 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.661953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.662092 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.662171 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.662942 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.663014 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 04:52:41.665467 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.654061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44971 (local address 127.0.0.1:41024)
0928 04:52:41.654508 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 04:52:41.654530 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 04:52:41.654578 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 04:52:41.657386 (+ 2808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 04:52:41.657387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 04:52:41.657509 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 04:52:41.657548 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.657549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.658433 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.658434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.659224 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 04:52:41.659225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.659688 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.659689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 04:52:41.659759 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.660120 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 04:52:41.660125 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 04:52:41.661468 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 04:52:41.664127 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.664128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.664128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.664325 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.664617 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 04:52:41.664618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 04:52:41.664619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 04:52:41.664688 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 04:52:41.665153 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 04:52:41.665154 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 04:52:41.665244 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 04:52:41.665294 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 04:52:41.665382 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430}
I20260928 04:52:41.665639 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18c3d522ae7b4f618de650b3ba641166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } } }
I20260928 04:52:41.665722 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:41.666086 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18c3d522ae7b4f618de650b3ba641166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } } }
I20260928 04:52:41.666185 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:52:41.666217 23241 mvcc.cc:205] Tried to move back new op lower bound from 7334179478133043200 to 7334179478070161408. Current Snapshot: MvccSnapshot[applied={T|T < 7334179478133043200}]
I20260928 04:52:41.667052 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:41.654571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.0.98.60:44971)
0928 04:52:41.656744 (+ 2173us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.656746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.656754 (+ 8us) server_negotiation.cc:407] Connection header received
0928 04:52:41.656785 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.656786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.656797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.656859 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.657613 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.658362 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.659295 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:41.659587 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:41.659643 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:41.661541 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 04:52:41.661543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 04:52:41.661543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 04:52:41.661549 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 04:52:41.663840 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.664405 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.664406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.664406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.664494 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 04:52:41.664777 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 04:52:41.664777 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 04:52:41.664777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 04:52:41.665029 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 04:52:41.665908 (+ 879us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:41.666815 (+ 907us) server_negotiation.cc:299] Negotiation successful
0928 04:52:41.666954 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2153}
I20260928 04:52:41.668368 23122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e: Bootstrap starting.
I20260928 04:52:41.669062 23122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e: Neither blocks nor log segments found. Creating new log.
I20260928 04:52:41.669401 23122 log.cc:824] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e: Log is configured to *not* fsync() on all Append() calls
I20260928 04:52:41.670271 23122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e: No bootstrap required, opened a new log
I20260928 04:52:41.672371 23122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } }
I20260928 04:52:41.672508 23122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c99e194a5d34b13a128cc0daae5de2e, State: Initialized, Role: FOLLOWER
I20260928 04:52:41.672590 23122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } }
I20260928 04:52:41.672896 23122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 04:52:41.672945 23096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18c3d522ae7b4f618de650b3ba641166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2c99e194a5d34b13a128cc0daae5de2e"
I20260928 04:52:41.673013 23096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 04:52:41.673900 23096 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18c3d522ae7b4f618de650b3ba641166 in term 1.
I20260928 04:52:41.674099 23095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [term 1 FOLLOWER]: Refusing update from remote peer 18c3d522ae7b4f618de650b3ba641166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 04:52:41.674553 23217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 04:52:41.674868 23232 catalog_manager.cc:1347] Generated new cluster ID: 71cd503849a74d138a086d1af7f0b96e
I20260928 04:52:41.674909 23232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 04:52:41.672838 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } } }
I20260928 04:52:41.675890 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:52:41.675972 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18c3d522ae7b4f618de650b3ba641166. Latest consensus state: current_term: 1 leader_uuid: "18c3d522ae7b4f618de650b3ba641166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } } }
I20260928 04:52:41.676054 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 04:52:41.676111 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18c3d522ae7b4f618de650b3ba641166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } } }
I20260928 04:52:41.676182 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e [sys.catalog]: This master's current role is: FOLLOWER
W20260928 04:52:41.678990 23257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 04:52:41.679075 23257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 04:52:41.679872 23259 mvcc.cc:205] Tried to move back new op lower bound from 7334179478133043200 to 7334179478070161408. Current Snapshot: MvccSnapshot[applied={T|T < 7334179478133043200}]
I20260928 04:52:41.689778 23217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18c3d522ae7b4f618de650b3ba641166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c3d522ae7b4f618de650b3ba641166" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35763 } } peers { permanent_uuid: "2c99e194a5d34b13a128cc0daae5de2e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42467 } } peers { permanent_uuid: "529fdd30500d4296adcf61b1047c8714" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44971 } } }
I20260928 04:52:41.690018 23217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166 [sys.catalog]: This master's current role is: LEADER
I20260928 04:52:41.692384 23232 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 04:52:41.693089 23232 catalog_manager.cc:1504] Loading token signing keys...
I20260928 04:52:41.698702 23232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18c3d522ae7b4f618de650b3ba641166: Generated new TSK 0
I20260928 04:52:41.698868 23232 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 04:52:41.724820 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.723266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.0.98.62:35763)
0928 04:52:41.723310 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.723312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.724212 (+ 900us) server_negotiation.cc:407] Connection header received
0928 04:52:41.724311 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.724312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.724324 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.724354 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.724755 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41434: BlockingRecv error: recv got EOF from 127.0.0.1:41434 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 04:52:41.725801 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.723924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.0.98.61:42467)
0928 04:52:41.723963 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.723965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.725165 (+ 1200us) server_negotiation.cc:407] Connection header received
0928 04:52:41.725319 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.725320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.725330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.725358 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.725744 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49718: BlockingRecv error: recv got EOF from 127.0.0.1:49718 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 04:52:41.726303 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.723592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.0.98.60:44971)
0928 04:52:41.723630 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.723632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.725052 (+ 1420us) server_negotiation.cc:407] Connection header received
0928 04:52:41.725418 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.725419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.725429 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.725456 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.726244 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41026: BlockingRecv error: recv got EOF from 127.0.0.1:41026 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 04:52:41.728420 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.727748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49722 (local address 127.0.98.61:42467)
0928 04:52:41.727779 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.727781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.727977 (+ 196us) server_negotiation.cc:407] Connection header received
0928 04:52:41.728072 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.728073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.728088 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.728115 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.728381 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49722: BlockingRecv error: recv got EOF from 127.0.0.1:49722 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 04:52:41.731191 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.730482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.0.98.61:42467)
0928 04:52:41.730511 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.730513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.730727 (+ 214us) server_negotiation.cc:407] Connection header received
0928 04:52:41.730814 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.730815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.730823 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.730849 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.731130 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49728: BlockingRecv error: recv got EOF from 127.0.0.1:49728 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 04:52:41.735046 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.734171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49732 (local address 127.0.98.61:42467)
0928 04:52:41.734221 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.734222 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.734494 (+ 272us) server_negotiation.cc:407] Connection header received
0928 04:52:41.734621 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.734622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.734631 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.734660 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.734998 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49732: BlockingRecv error: recv got EOF from 127.0.0.1:49732 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 04:52:41.741127 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.740322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.0.98.61:42467)
0928 04:52:41.740356 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.740376 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.740622 (+ 246us) server_negotiation.cc:407] Connection header received
0928 04:52:41.740716 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.740717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.740726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.740755 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.741081 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49746: BlockingRecv error: recv got EOF from 127.0.0.1:49746 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 04:52:41.751294 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.750403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.0.98.61:42467)
0928 04:52:41.750457 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.750460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.750721 (+ 261us) server_negotiation.cc:407] Connection header received
0928 04:52:41.750818 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.750819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.750829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.750857 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.751210 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49760: BlockingRecv error: recv got EOF from 127.0.0.1:49760 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 04:52:41.769583 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.768517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.0.98.61:42467)
0928 04:52:41.768559 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.768561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.768851 (+ 290us) server_negotiation.cc:407] Connection header received
0928 04:52:41.768968 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.768969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.768980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.769026 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.769528 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49772: BlockingRecv error: recv got EOF from 127.0.0.1:49772 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 04:52:41.803740 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.802974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.0.98.61:42467)
0928 04:52:41.803024 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.803026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.803150 (+ 124us) server_negotiation.cc:407] Connection header received
0928 04:52:41.803191 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.803193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.803205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.803296 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.803679 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49782: BlockingRecv error: recv got EOF from 127.0.0.1:49782 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 04:52:41.869776 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.868935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.0.98.61:42467)
0928 04:52:41.869002 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.869005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.869165 (+ 160us) server_negotiation.cc:407] Connection header received
0928 04:52:41.869279 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.869280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.869291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.869333 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.869699 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49784: BlockingRecv error: recv got EOF from 127.0.0.1:49784 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 04:52:41.999657 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:41.998886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.0.98.61:42467)
0928 04:52:41.998988 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:41.998991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:41.999105 (+ 114us) server_negotiation.cc:407] Connection header received
0928 04:52:41.999192 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:41.999192 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:41.999204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:41.999275 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:41.999594 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49790: BlockingRecv error: recv got EOF from 127.0.0.1:49790 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 04:52:42.257501 23126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 04:52:42.256766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.0.98.61:42467)
0928 04:52:42.256847 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:42.256850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:42.256860 (+ 10us) server_negotiation.cc:407] Connection header received
0928 04:52:42.256957 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:42.256958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:42.256970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:42.257059 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 04:52:42.257438 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49802: BlockingRecv error: recv got EOF from 127.0.0.1:49802 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 04:52:42.601616 23214 catalog_manager.cc:1259] Loaded cluster ID: 71cd503849a74d138a086d1af7f0b96e
I20260928 04:52:42.601680 23214 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714: loading cluster ID for follower catalog manager: success
I20260928 04:52:42.602419 23214 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714: acquiring CA information for follower catalog manager: success
I20260928 04:52:42.602672 23214 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 529fdd30500d4296adcf61b1047c8714: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:52:42.679507 23257 catalog_manager.cc:1259] Loaded cluster ID: 71cd503849a74d138a086d1af7f0b96e
I20260928 04:52:42.679579 23257 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e: loading cluster ID for follower catalog manager: success
I20260928 04:52:42.680408 23257 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e: acquiring CA information for follower catalog manager: success
I20260928 04:52:42.680722 23257 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c99e194a5d34b13a128cc0daae5de2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 04:52:42.773057 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:42.770591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.0.98.61:42467)
0928 04:52:42.770717 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:42.770719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:42.770761 (+ 42us) server_negotiation.cc:407] Connection header received
0928 04:52:42.770823 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:42.770823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:42.770873 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:42.770945 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:42.771085 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:42.771552 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:42.772275 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:42.772465 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:42.772515 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:42.772648 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:42.772811 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:42.772912 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:42.772927 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 04:52:42.772945 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260928 04:52:42.776979 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:42.774928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.0.98.60:44971)
0928 04:52:42.775072 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:42.775074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:42.775089 (+ 15us) server_negotiation.cc:407] Connection header received
0928 04:52:42.775140 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:42.775140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:42.775171 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:42.775237 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:42.775317 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:42.775769 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:42.776426 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:42.776602 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:42.776648 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:42.776748 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:42.776830 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:42.776876 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:42.776910 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 04:52:42.776924 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260928 04:52:42.781076 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:42.778487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.0.98.61:42467)
0928 04:52:42.778530 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:42.778532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:42.778716 (+ 184us) server_negotiation.cc:407] Connection header received
0928 04:52:42.778781 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:42.778782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:42.778791 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:42.778855 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:42.778990 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:42.779441 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:42.780276 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:42.780566 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:42.780627 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:42.780786 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:42.780816 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:42.780864 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:42.780961 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 04:52:42.780983 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 04:52:42.781653 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:42.778757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.0.98.60:44971)
0928 04:52:42.778900 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:42.778901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:42.778989 (+ 88us) server_negotiation.cc:407] Connection header received
0928 04:52:42.779094 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:42.779096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:42.779105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:42.779131 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:42.779277 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:42.779932 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:42.780843 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:42.781163 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:42.781246 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:42.781402 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:42.781431 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:42.781481 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:42.781553 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 04:52:42.781571 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260928 04:52:42.781749 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 04:52:42.778489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.0.98.62:35763)
0928 04:52:42.778730 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 04:52:42.778733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 04:52:42.778740 (+ 7us) server_negotiation.cc:407] Connection header received
0928 04:52:42.778780 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 04:52:42.778780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 04:52:42.778825 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 04:52:42.778904 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 04:52:42.779050 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:42.779831 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:42.780619 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 04:52:42.780900 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 04:52:42.780962 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 04:52:42.781225 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 04:52:42.781430 (+ 205us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 04:52:42.781485 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 04:52:42.781629 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 04:52:42.781651 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":53,"threads_started":1}
I20260928 04:52:42.786980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 22979
I20260928 04:52:42.792173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 23054
I20260928 04:52:42.796685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNuqiey/build/release/bin/kudu with pid 23129
2026-09-28T04:52:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2771 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5568 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (265291 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST